Causal Imitation Learning with Unobserved Confounders

Introduction

A unifying theme of Artificial Intelligence is to learn a policy from observations in an unknown environment such that a suitable level of performance is achieved [33, Ch. 1.1]. Operationally, a policy is a decision rule that determines an action based on a certain set of covariates; observations are possibly generated by a human demonstrator following a different behavior policy. The task of evaluating policies from a combination of observational data and assumptions about the underlying environment has been studied in the literature of causal inference and reinforcement learning . Several criteria, algorithms, and estimation methods have been developed to solve this problem . In many applications, it is not clear which performance measure the demonstrator is (possibly subconsciously) optimizing. That is, the reward signal is not labeled and accessible in the observed expert’s trajectories. In such settings, the performance of candidate policies is not uniquely discernible from the observational data due to latent outcomes, even when infinitely many samples are gathered, complicating efforts to learn policy with satisfactory performance.

An alternative approach used to circumvent this issue is to find a policy that mimics a demonstrator’s behavior, which leads to the imitation learning paradigm . The expectation (or rather hope) is that if the demonstrations are generated by an expert with near-optimal reward, the performance of the imitator would also be satisfactory. Current methods of imitation learning can be categorized into behavior cloning and inverse reinforcement learning . The former focuses on learning a nominal expert policy that approximates the conditional distribution mapping observed input covariates of the behavior policy to the action domain. The latter attempts to learn a reward function that prioritizes observed behaviors of the expert; reinforcement learning methods are then applied using the learned reward function to obtain a nominal policy. However, both families of methods rely on the assumption that the expert’s input observations match those available to the imitator. When unobserved covariates exist, however, naively imitating the nominal expert policy does not necessarily lead to a satisfactory performance, even when the expert him or herself behaves optimally.

This example shows that even when one is able to perfectly mimic an optimal demonstrator, the learned policy can still be suboptimal. In this paper, we try to explicate this phenomenon and, more broadly, understand imitability through a causal lens Some recent progress in the field of causal imitation has been reported, albeit oblivious to the phenomenon described above and our contributions. Some work considered settings in which the input to the expert policy is fully observed , while another assumed that the primary outcome is observed (e.g., $Y$ in Figure 1(a)) . . Our task is to learn an imitating policy that achieves the expert’s performance from demonstration data in a structural causal model [29, Ch. 7], allowing for unobserved confounders (UCs) affecting both action and outcome variables. Specifically, our contributions are summarized as follows. (1) We introduce a complete graphical criterion for determining the feasibility of imitation from demonstration data and qualitative knowledge about the data-generating process represented as a causal graph. (2) We develop a sufficient algorithm for identifying an imitating policy when the given criterion does not hold, by leveraging the quantitative knowledge in the observational distribution. (3) We provide an efficient and practical procedure for finding an imitating policy through explicit parametrization of the causal model, and use it to validate our results on high-dimensional, synthetic datasets. For the sake of space constraints, we provide all proofs in the complete technical report [15, Appendix A].

In this section, we introduce the basic notations and definitions used throughout the paper. We use capital letters to denote random variables ($X$) and small letters for their values ($x$). $\mathscr{D}_{X}$ represents the domain of $X$ and $\mathscr{P}_{X}$ the space of probability distributions over $\mathscr{D}_{X}$. For a set $\bm{X}$, $|\bm{X}|$ denotes its dimension. We consistently use the abbreviation $P(x)$ to represent the probabilities $P(X=x)$. Finally, $I_{\{\bm{Z}=\bm{z}\}}$ is an indicator function that returns $1$ if $\bm{Z}=\bm{z}$ holds true; otherwise $0$.

Calligraphic letters, e.g., $\mathcal{G}$, will be used to represent directed acyclic graphs (DAGs) (e.g., Figure 1). We denote by $\mathcal{G}_{\overline{\bm{X}}}$ the subgraph obtained from $\mathcal{G}$ by removing arrows coming into nodes in $\bm{X}$; $\mathcal{G}_{\underline{\bm{X}}}$ is a subgraph of $\mathcal{G}$ by removing arrows going out of $\bm{X}$. We will use standard family conventions for graphical relationships such as parents, children, descendants, and ancestors. For example, the set of parents of $\bm{X}$ in $\mathcal{G}$ is denoted by $\mathit{pa}(\bm{X})_{\mathcal{G}}=\cup_{X\in\bm{X}}\mathit{pa}(X)_{\mathcal{G}}$. $\mathit{ch}$, $\mathit{de}$ and $\mathit{an}$ are similarly defined, We write $\mathit{Pa},\mathit{Ch},\mathit{De},\mathit{An}$ if arguments are included as well, e.g. $\mathit{De}(\bm{X})_{\mathcal{G}}=\mathit{de}(\bm{X})_{\mathcal{G}}\cup\bm{X}$. A path from a node $X$ to a node $Y$ in $\mathcal{G}$ is a sequence of edges which does not include a particular node more than once. Two sets of nodes $\bm{X},\bm{Y}$ are said to be d-separated by a third set $\bm{Z}$ in a DAG $\mathcal{G}$, denoted by $(\bm{X}\perp\!\!\!\!\perp\bm{Y}|\bm{Z})_{\mathcal{G}}$, if every edge path from nodes in one set to nodes in another are “blocked”. The criterion of blockage follows [29, Def. 1.2.3].

The basic semantic framework of our analysis rests on structural causal models (SCMs) [29, Ch. 7]. An SCM $M$ is a tuple $\langle\bm{U},\bm{V},\mathcal{F},P(\bm{u})\rangle$ where $\bm{V}$ is a set of endogenous variables and $\bm{U}$ is a set of exogenous variables. $\mathcal{F}$ is a set of structural functions where $f_{V}\in\mathcal{F}$ decides values of an endogenous variable $V\in\bm{V}$ taking as argument a combination of other variables. That is, $V\leftarrow f_{V}(\mathit{Pa}_{V},U_{V}),\mathit{Pa}_{V}\subseteq\bm{V},U_{V}\subseteq\bm{U}$. Values of $\bm{U}$ are drawn from an exogenous distribution $P(\bm{u})$. Each SCM $M$ induces a distribution $P(\bm{v})$ over endogenous variables $\bm{V}$. An intervention on a subset $\bm{X}\subseteq\bm{V}$, denoted by $\text{do}(\bm{x})$, is an operation where values of $\bm{X}$ are set to constants $\bm{x}$, replacing the functions $\{f_{X}:\forall X\in\bm{X}\}$ that would normally determine their values. For an SCM $M$, let $M_{\bm{x}}$ be a submodel of $M$ induced by intervention $\text{do}(\bm{x})$. For a set $\bm{S}\subseteq\bm{V}$, the interventional distribution $P(\bm{s}|\text{do}(\bm{x}))$ induced by $\text{do}(\bm{x})$ is defined as the distribution over $\bm{S}$ in the submodel $M_{\bm{x}}$, i.e., $P(\bm{s}|\text{do}(\bm{x});M)\triangleq P(\bm{s};M_{\bm{x}})$. We leave $M$ implicit when it is obvious from the context. For a detailed survey on SCMs, we refer readers to [29, Ch. 7].

Imitation Learning in Structural Causal Models

In this section, we formalize and study the imitation learning problem in causal language. We first define a special type of SCM that explicitly allows one to model the unobserved nature of some endogenous variables, which is called the partially observable structural causal model (POSCM). This definition will facilitate the more explicitly articulation of which endogenous variables are available to the demonstrator and corresponding policy at each point in time.

A POSCM is a tuple $\langle M,\bm{O},\bm{L}\rangle$, where $M$ is a SCM $\langle\bm{U},\bm{V},\mathcal{F},P(\bm{u})\rangle$ and $\langle\bm{O},\bm{L}\rangle$ is a pair of subsets forming a partition over $\bm{V}$ (i.e., $\bm{V}=\bm{O}\cup\bm{L}$ and $\bm{O}\cap\bm{L}=\emptyset$); $\bm{O}$ and $\bm{L}$ are called observed and latent endogenous variables, respectively.

Each POSCM $M$ induces a probability distribution over $V$, of which one can measure the observed variables $\bm{O}$. $P(\bm{o})$ is usually called the observational distribution. $M$ is associated with a causal diagram $\mathcal{G}$ (e.g., see Figure 1) where solid nodes represent observed variables $\bm{O}$, dashed nodes represent latent variables $\bm{L}$, and arrows represent the arguments $\mathit{Pa}_{V}$ of each functional relationship $f_{V}$. Exogenous variables $\bm{U}$ are not explicitly shown; a bi-directed arrow between nodes $V_{i}$ and $V_{j}$ indicates the presence of an unobserved confounder (UC) affecting both $V_{i}$ and $V_{j}$, i.e., $U_{V_{i}}\cap U_{V_{j}}\neq\emptyset$.

Consider a POSCM $\langle M,\bm{O},\bm{L}\rangle$ with $M=\langle\bm{U},\bm{V},\mathcal{F},P(\bm{u})\rangle$. Our goal is to learn an efficient policy to decide the value of an action variable $X\in\bm{O}$. The performance of the policy is evaluated using the expected value of a reward variable $Y$. Throughout this paper, we assume that reward $Y$ is latent and $X$ affects $Y$ (i.e., $Y\in\bm{L}\cap\mathit{De}(X)_{\mathcal{G}}$). A policy $\pi$ is a function mapping from values of covariates $\mathit{Pa}^{*}\subseteq\bm{O}\setminus\mathit{De}(X)_{\mathcal{G}_{\overline{X}}}$ $\mathcal{G}_{\overline{X}}$ is a causal diagram associated with the submodel $M_{x}$ induced by intervention $do(x)$. to a probability distribution over $X$, which we denote by $\pi(x|\mathit{pa}^{*})$. An intervention following a policy $\pi$, denoted by $\text{do}(\pi)$, is an operation that draws values of X independently following $\pi$, regardless of its original (natural) function $f_{X}$. Let $M_{\pi}$ denote the manipulated SCM of $M$ induced by $\text{do}(\pi)$. Similar to atomic settings, the interventional distribution $P(\bm{v}|\text{do}(\pi))$ is defined as the distribution over $\bm{V}$ in the manipulated model $M_{\pi}$, given by,

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, let $\bm{Y}$ be an arbitrary subset of $\bm{V}$. $P(\bm{y}|\text{do}(\pi))$ is said to be identifiable w.r.t. $\langle\mathcal{G},\Pi\rangle$ if $P\left(\bm{y}|\text{do}(\pi);M\right)$ is uniquely computable from $P(\bm{o};M)$ and $\pi$ for any POSCM $M\in\mathscr{M}_{\langle\mathcal{G}\rangle}$ and any $\pi\in\Pi$.

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, let $\bm{Y}$ be an arbitrary subset of $\bm{V}$. If not all variables in $\bm{Y}$ are observed (i.e., $\bm{Y}\cap\bm{L}\neq\emptyset$), $P(\bm{y}|\text{do}(\pi))$ is not identifiable.

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, let $\bm{Y}$ be an arbitrary subset of $\bm{V}$. $P(\bm{y})$ is said to be imitable w.r.t. $\langle\mathcal{G},\Pi\rangle$ if there exists a policy $\pi\in\Pi$ uniquely computable from $P(\bm{o})$ such that $P(\bm{y}|\text{do}(\pi);M)=P(\bm{y};M)$ for any POSCM $M\in\mathscr{M}_{\langle\mathcal{G}\rangle}$.

Our task is to determine the imitability of the expert performance. More specifically, we want to learn an imitating policy $\pi\in\Pi$ from $P(\bm{o})$ such that $P(y|\text{do}(\pi))=P(y)$ The imitation is trivial if $Y\not\in\mathit{De}(X)_{\mathcal{G}}$: by Rule 3 of [29, Thm. 3.4.1] (or [6, Thm. 1]), $P(y|\text{do}(\pi))=P(y)$ for any policy $\pi$. This paper aims to find a specific $\pi$ satisfying $P(y|\text{do}(\pi))=P(y)$ even when $Y\in\mathit{De}(X)_{\mathcal{G}}$., in any POSCM $M$ associated with the causal diagram $\mathcal{G}$. Consider Figure 3(a) as an example. $P(y)$ is imitable with policy $\pi(x)=P(x)$ since by Equation 1 and marginalization, $P(y|\text{do}(\pi))=\sum_{x,w}P(y|w)P(w|x)\pi(x)=\sum_{x,w}P(y|w)P(w|x)P(x)=P(y)$. In practice, unfortunately, the expert’s performance cannot always be imitated. To understand this setting, we first write, more explicitly, the conditions under which this is not the case:

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, let $\bm{Y}$ be an arbitrary subset of $\bm{V}$. $P(\bm{y})$ is not imitable w.r.t. $\langle\mathcal{G},\Pi\rangle$ if there exists two POSCMs $M_{1},M_{2}\in\mathscr{M}_{\langle\mathcal{G}\rangle}$ satisfying $P(\bm{o};M_{1})=P(\bm{o};M_{2})$ while there exists no policy $\pi\in\Pi$ such that for $i=1,2$, $P(\bm{y}|\text{do}(\pi);M_{i})=P(\bm{y};M_{i})$.

It follows as a corollary that $P(\bm{y})$ is not imitable if there exists a POSCM $M$ compatible with $\mathcal{G}$ such that no policy $\pi\in\Pi$ could ensure $P(\bm{y}|\text{do}(\pi);M)=P(\bm{y};M)$. For instance, consider the causal diagram $\mathcal{G}$ and policy space $\Pi$ in Figure 3(b). Here, the expert’s reward $P(y)$ is not imitable: consider a POSCM with functions $X\leftarrow U,W\leftarrow X,Y\leftarrow U\oplus\neg W$; values $U$ are drawn uniformly over $\{0,1\}$. In this model, $P(Y=1|\text{do}(\pi))=0.5$ for any policy $\pi$, which is far from the optimal expert reward, $P(Y=1)=1$.

An interesting observation from the above example of Figure 3(b) is that the effect $P(y|\text{do}(\pi))$ is identifiable, following the front-door criterion in [29, Thm. 3.3.4], but no policy imitates the corresponding $P(y)$. However, in some settings, the expert’s reward $P(y)$ is imitable but the imitator’s reward $P(y|\text{do}(\pi))$ cannot be uniquely determined. To witness, consider again the example in Figure 3(a). The imitability of $P(y)$ has been previously shown; while $P(y|\text{do}(\pi))$ is not identifiable due to latent reward $Y$ (Corollary 1).

In general, the problem of imitability is orthogonal to identifiability, and, therefore, requires separate consideration. Since imitability does not always hold, we introduce a useful graphical criterion for determining whether imitating an expert’s performance is feasible, and if so, how.

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, $P(y)$ is imitable w.t.r. $\langle\mathcal{G},\Pi\rangle$ if $\mathit{pa}(X)_{\mathcal{G}}\subseteq\mathit{Pa}(\Pi)$ and there is no bi-directed arrow pointing to $X$ in $\mathcal{G}$. Moreoever, the imitating policy $\pi\in\Pi$ is given by $\pi\left(x|\mathit{pa}(\Pi)\right)=P\left(x|\mathit{pa}(X)_{\mathcal{G}}\right)$.

In words, Theorem 1 says that if the expert and learner share the same policy space, then the policy is always imitable. In fact, this result can be seen as a causal justification for when the method of “behavior cloning”, widely used in practice, is valid, leading to proper imitation. When the original behavior policy $f_{X}$ is contained in the policy space $\Pi$, the leaner could imitate the expert’s reward $P(y)$ by learning a policy $\pi\in\Pi$ that matches the distribution $P(x|\mathit{pa}(\Pi))$ . Next, we consider the more challenging setting when policy spaces of the expert and learner disagree (i.e., the learner and expert have different views of the world, $f_{X}\not\in\Pi$). We will leverage a graphical condition adopted from the celebrated backdoor criterion [29, Def. 3.3.1].

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, a set $\bm{Z}$ is said to satisfy the $\pi$-backdoor criterion w.r.t. $\langle\mathcal{G}.\Pi\rangle$ if and only if $\bm{Z}\subseteq\mathit{Pa}(\Pi)$ and $(Y\perp\!\!\!\!\perp X|\bm{Z})_{\mathcal{G}_{\underline{X}}}$, which is called the $\pi$-backdoor admissible set w.r.t. $\langle\mathcal{G},\Pi\rangle$.

For concreteness, consider again the highway driving example in Figure 1(a). There exists no $\pi$-backdoor admissible set due to the path $X\leftarrow L\rightarrow Y$. Now consider a modified graph in Figure 1(b) where edge $L\rightarrow Y$ is removed. $\{Z\}$ is $\pi$-backdoor admissible since $Z\in\mathit{Pa}(\Pi)$ and $\left(Y\perp\!\!\!\!\perp X|Z\right)_{\mathcal{G}_{\underline{X}}}$. Leveraging the imitation backdoor condition, our next theorem provides a full characterization for when imitating expert’s performance is achievable, despite the fact that the reward $Y$ is latent.

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, $P(y)$ is imitable w.r.t. $\langle\mathcal{G},\Pi\rangle$ if and only if there exists an $\pi$-backdoor admissible set $\bm{Z}$ w.r.t. $\langle\mathcal{G},\Pi\rangle$. Moreover, the imitating policy $\pi\in\Pi$ is given by $\pi\left(x|\mathit{pa}(\Pi)\right)=P\left(x|\bm{z}\right)$.

That is, one can learn an imitating policy from a policy space $\Pi^{\prime}=\{\pi:\mathscr{D}_{\bm{Z}}\mapsto\mathscr{P}_{X}\}$ that mimics the conditional probabilities $P(x|\bm{z})$ if and only if $\bm{Z}$ is $\pi$-backdoor admissible. If that is the case, such a policy can be learned from data through standard density estimation methods. For instance, Theorem 2 ascertains that $P(y)$ in Figure 1(a) is indeed non-imitable. On the other hand, $P(y)$ in Figure 1(b) is imitable, guaranteed by the $\pi$-backdoor admissible set $\{Z\}$; the imitating policy is given by $\pi(x|z)=P(x|z)$.

Causal Imitation Learning with Data Dependency

One may surmise that the imitation boundary established by Theorem 2 suggests that when there exists no $\pi$-backdoor admissible set, it is infeasible to imitate the expert performance from observed trajectories of demonstrations. In this section, we will circumvent this issue by exploiting actual parameters of the observational distribution $P(\bm{o})$. In particular, we denote by $\mathscr{M}_{\langle\mathcal{G},P\rangle}$ a subfamily of candidate models in $\mathscr{M}_{\langle\mathcal{G}\rangle}$ that induce both the causal diagram $\mathcal{G}$ and the observational distribution $P(\bm{o})$, i.e., $\mathscr{M}_{\langle\mathcal{G},P\rangle}=\left\{\forall M\in\mathscr{M}_{\langle\mathcal{G}\rangle}:P(\bm{o};M)=P(\bm{o})\right\}$. We introduce a refined notion of imitability that will explore the quantitative knowledge of observations $P(\bm{o})$ (to be exemplified). Formally,

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$, and an observational distribution $P(\bm{o})$, let $\bm{Y}$ be an arbitrary subset of $\bm{V}$. $P(\bm{y})$ is said to be practically imitable (for short, p-imitable) w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$ if there exists a policy $\pi\in\Pi$ uniquely computable from $P(\bm{o})$ such that $P(\bm{y}|\text{do}(\pi);M)=P(\bm{y};M)$ for any POSCM $M\in\mathscr{M}_{\langle\mathcal{G},P\rangle}$.

The following corollary can be derived based on the definition of practical imitability.

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$ and an observational distribution $P(\bm{o})$, let a subset $\bm{Y}\subseteq\bm{V}$. If $P(\bm{y})$ is imitable w.r.t. $\langle\mathcal{G},\Pi\rangle$, $P(\bm{y})$ is p-imitable w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$.

Compared to Definition 3, the practical imitability of Definition 5 aims to find an imitating policy for a subset of candidate POSCMs $\mathscr{M}_{\langle\mathcal{G},P\rangle}$ restricted to match a specific observational distribution $P(\bm{o})$. Definition 3, on the other hand, requires only the causal diagram $\mathcal{G}$. In other words, for an expert’s performance $P(y)$ that is non-imitable w.r.t. $\langle\mathcal{G},\Pi\rangle$, it could still be p-imitable after analyzing actual probabilities of the observational distribution $P(\bm{o})$.

For concreteness, consider again $P(y)$ in Figure 3(b) which is not imitable due to the bi-directed arrow $X\leftrightarrow Y$. However, new imitation opportunities arise when actual parameters of the observational distribution $P(x,w,y)$ are provided. Suppose the underlying POSCM is given by: $X\leftarrow U_{X}\oplus U_{Y}$, $W\leftarrow X\oplus U_{W}$, $Y\leftarrow W\oplus U_{Y}$ where $U_{X},U_{Y},U_{W}$ are independent binary variables drawn from $P(U_{X}=1)=P(U_{Y}=1)=P(U_{W}=0)=0.9$. Here, the causal effect $P(y|\text{do}(x))$ is identifiable from $P(x,w,y)$ following the front-door formula $P(y|\text{do}(x))=\sum_{w}P(w|x)\sum_{x^{\prime}}P(y|w,x^{\prime})P(x^{\prime})$ [29, Thm. 3.3.4]. We thus have $P(Y=1|\text{do}(X=0))=0.82$ which coincides with $P(Y=1)=0.82$, i.e., $P(y)$ is p-imitable with atomic intervention $\text{do}(X=0)$. In the most practical settings, the expert reward $P(y)$ rarely equates to $P(y|\text{do}(x))$; stochastic policies $\pi(x)$ are then applicable to imitate $P(y)$ by re-weighting $P(y|\text{do}(x))$ induced by the corresponding atomic interventions Consider a variation of the model where $P(U_{W}=1)=0.7$. $P(y)$ is p-imitable with $\pi(X=0)=0.75$.. To tackle p-imitability in a general way, we proceed by defining a set of observed variables that serve as a surrogate of the unobserved $Y$ with respect to interventions on $X$. Formally,

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$, let $\bm{S}$ be an arbitrary subset of $\bm{O}$. $\bm{S}$ is an imitation surrogate (for short, surrogate) w.r.t. $\langle\mathcal{G},\Pi\rangle$ if $(Y\perp\!\!\!\!\perp\hat{X}|\bm{S})_{\mathcal{G}\cup\Pi}$ where $\mathcal{G}\cup\Pi$ is a supergraph of $\mathcal{G}$ by adding arrows from $\mathit{Pa}(\Pi)$ to $X$; $\hat{X}$ is a new parent to $X$.

An surrogate $\bm{S}$ is said to be minimal if there exists no subset $\bm{S}^{\prime}\subset\bm{S}$ such that $\bm{S}^{\prime}$ is also a surrogate w.r.t. $\langle\mathcal{G},\Pi\rangle$. Consider as an example Figure 1(c) where the supergraph $\mathcal{G}\cup\Pi$ coincides with the causal diagram $\mathcal{G}$. By Definition 6, both $\{W,S\}$ and $\{S\}$ are valid surrogate relative to $\langle X,Y\rangle$ with $\{S\}$ being the minimal one. By conditioning on $S$, the decomposition of Equation 1 implies $P(y|\text{do}(\pi))=\sum_{s,w,u}P(y|s)P(s|w,u)P(w|x)\pi(x)P(u)=\sum_{s}P(y|s)P(s|\text{do}(\pi))$. That is, the surrogate $S$ mediates all influence of interventions on action $X$ to reward $Y$. It is thus sufficient to find an imitating policy $\pi$ such that $P(s|\text{do}(\pi))=P(s)$ for any POSCM $M$ associated with Figure 1(c). The resultant policy is guaranteed to imitate the expert’s reward $P(y)$.

When a surrogate $\bm{S}$ is found and $P(\bm{s}|\text{do}(\pi))$ is identifiable, one could compute $P(\bm{s}|\text{do}(\pi))$ for each policy $\pi$ and check if it matches $P(\bm{s})$. In many settings, however, $P(\bm{s}|\text{do}(\pi))$ is not identifiable w.r.t. $\langle\mathcal{G},\Pi\rangle$. For example, in Figure 1(d), $S$ is a surrogate w.r.t. $\langle\mathcal{G},\Pi\rangle$, but $P(s|\text{do}(\pi))$ is not identifiable due to collider $Z$ ($\pi$ uses non-descendants as input by default). Fortunately, identifying $P(\bm{s}|\text{do}(\pi))$ may still be feasible in some subspaces of $\Pi$:

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$, and a subset $\bm{S}\subseteq\bm{O}$, let $\Pi^{\prime}$ be a policy subspace of $\Pi$. $\Pi^{\prime}$ is said to be an identifiable subspace (for short, id-subspace) w.r.t. $\langle\mathcal{G},\Pi,\bm{S}\rangle$ if $P(\bm{s}|\text{do}(\pi))$ is identifiable w.r.t. $\langle\mathcal{G},\Pi^{\prime}\rangle$.

Consider a policy subspace $\Pi^{\prime}=\{\pi:\mathscr{P}_{X}\}$ described in Figure 1(d) (i.e. $\pi$ that does not exploit information from covariates $Z$). $P(s|\text{do}(\pi))$ is identifiable w.r.t. $\langle\mathcal{G},\Pi^{\prime}\rangle$ following the front-door adjustment on $W$ [29, Thm. 3.3.4]. We could then evaluate interventional probabilities $P(s|\text{do}(\pi))$ for each policy $\pi\in\Pi^{\prime}$ from the observational distribution $P(x,w,s,z)$; the imitating policy is obtainable by solving the equation $P(s|\text{do}(\pi))=P(s)$. In other words, $\{S\}$ and $\Pi^{\prime}$ forms an instrument that allows one to solve the imitation learning problem in Figure 1(d).

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, let $\bm{S}$ be a subset of $\bm{O}$ and $\Pi^{\prime}$ be a subspace of $\Pi$. $\langle\bm{S},\Pi^{\prime}\rangle$ is said to be an imitation instrument (for short, instrument) if $\bm{S}$ is a surrogate w.r.t. $\langle\mathcal{G},\Pi^{\prime}\rangle$ and $\Pi^{\prime}$ is an id-subspace w.r.t. $\langle\mathcal{G},\Pi,\bm{S}\rangle$.

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$, and an observational distribution $P(\bm{o})$, let $\langle\bm{S},\Pi^{\prime}\rangle$ be an instrument w.r.t. $\langle\mathcal{G},\Pi\rangle$. If $P(\bm{s})$ is p-imitable w.r.t. $\langle\mathcal{G},\Pi^{\prime},P(\bm{o})\rangle$, then $P(y)$ is p-imitable w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$. Moreover, an imitating policy $\pi$ for $P(\bm{s})$ w.r.t. $\langle\mathcal{G},\Pi^{\prime},P(\bm{o})\rangle$ is also imitating policy for $P(y)$ w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$.

In words, Lemma 2 shows that when an imitation instrument $\langle\bm{S},\Pi^{\prime}\rangle$ is present, we could reduce the original imitation learning on a latent reward $Y$ to a p-imitability problem over observed surrogate variables $\bm{S}$ using policies in an identifiable subspace $\Pi^{\prime}$. The imitating policy $\pi$ is obtainable by solving the equation $P(\bm{s}|\text{do}(\pi))=P(\bm{s})$.

Our task in this section is to introduce a general algorithm that finds instruments, and learns a p-imitating policy given $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$. A naïve approach is to enumerate all pairs of subset $\bm{S}$ and subspace $\Pi^{\prime}$ and check whether they form an instrument; if so, we can compute an imitating policy for $P(\bm{s})$ w.r.t. $\langle\mathcal{G},\Pi^{\prime},P(\bm{o})\rangle$. However, the challenge is that the number of all possible subspaces $\Pi^{\prime}$ (or subsets $\bm{S}$) can be exponentially large. Fortunately, we can greatly restrict this search space. Let $\mathcal{G}\cup\{Y\}$ denote a causal diagram obtained from $\mathcal{G}$ by making reward $Y$ observed. The following proposition suggests that it suffices to consider only identifiable subspaces w.r.t. $\langle\mathcal{G}\cup\{Y\},\Pi,Y\rangle$.

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$, let a subspace $\Pi^{\prime}\subseteq\Pi$. If there exists $\bm{S}\subseteq\bm{O}$ such that $\langle\bm{S},\Pi^{\prime}\rangle$ is an instrument w.r.t. $\langle\mathcal{G},\Pi\rangle$, $\Pi^{\prime}$ is an id-subspace w.r.t. $\langle\mathcal{G}\cup\{Y\},\Pi,Y\rangle$.

Our algorithm Imitate is described in Algorithm 1. We assume access to an Identify oracle that takes as input a causal diagram $\mathcal{G}$, a policy space $\Pi$ and a set of observed variables $\bm{S}$. If $P(\bm{s}|\text{do}(\pi))$ is identifiable w.r.t. $\langle\mathcal{G},\Pi\rangle$, Identify returns “Yes”; otherwise, it returns “No”. For details about the Identify oracle, we refer readers to [15, Appendix B]. More specifically, Imitate takes as input a causal diagram $\mathcal{G}$, a policy space $\Pi$ and an observational distribution $P(\bm{o})$. At Step 2, Imitate applies a subroutine ListIdSpace to list identifiable subspaces $\Pi^{\prime}$ w.r.t. $\langle\mathcal{G}\cup\{Y\},\Pi,Y\rangle$, following the observation made in Lemma 3. The implementation details of ListIdSpace are provided in [15, Appendix C]. When an identifiable subspace $\Pi^{\prime}$ is found, Imitate tries to obtain a surrogate $\bm{S}$ w.r.t the diagram $\mathcal{G}$ and subspace $\Pi^{\prime}$. While there could exist multiple such surrogates, the following proposition shows that it is sufficient to consider only minimal ones.

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$, an observational distribution $P(\bm{o})$ and a subset $\bm{S}\subseteq\bm{O}$. $P(\bm{s})$ is p-imitable only if for any $\bm{S}^{\prime}\subseteq\bm{S}$, $P(\bm{s}^{\prime})$ is p-imitable w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$.

We apply a subroutine ListMinSep in to enumerate minimal surrogates in $\bm{O}$ that d-separate $\hat{X}$ and $Y$ in the supergraph $\mathcal{G}\cup\Pi^{\prime}$. When a minimal surrogate $\bm{S}$ is found, Imitate uses the Identify oracle to validate if $P(\bm{s}|\text{do}(\pi))$ is identifiable w.r.t. $\langle\mathcal{G},\Pi^{\prime}\rangle$, i.e., $\langle\bm{S},\Pi^{\prime}\rangle$ form an instrument. Consider Figure 1(d) as an example. While $P(y|\text{do}(\pi))$ is not identifiable for every policy in $\Pi$ had $Y$ been observed, $\Pi$ contains an id-subspace $\{\pi:\mathscr{P}_{X}\}$ w.r.t. $\langle\mathcal{G}\cup\{Y\},\Pi,Y\rangle$, which is associated with a minimal surrogate $\{S\}$. Applying Identify confirms that $\langle\{S\},\{\pi:\mathscr{P}_{X}\}\rangle$ is an instrument.

At Step 5, Imitate solves for a policy $\pi$ in the subspace $\Pi^{\prime}$ that imitates $P(\bm{s})$ for all instances in the hypothesis class $\mathscr{M}_{\langle\mathcal{G},P\rangle}$. If such a policy exists, Imitate returns $\pi$; otherwise, the algorithm continues. Since $\langle\bm{S},\Pi^{\prime}\rangle$ is an instrument, Lemma 2 implies that the learned policy $\pi$, if it exists, is ensured to imitate the expert reward $P(y)$ for any POSCM $M\in\mathscr{M}_{\langle\mathcal{G},P\rangle}$.

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$, and an observational distribution $P(\bm{o})$, if Imitate returns a policy $\pi\in\Pi$, $P(y)$ is p-imitable w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$. Moreover, $\pi$ is an imitating policy for $P(y)$ w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$.

2 Optimizing Imitating Policies

We now introduce optimization procedures to solve for an imitating policy at Step 5 of Imitate algorithm. Since the pair $\langle\bm{S},\Pi^{\prime}\rangle$ forms a valid instrument (ensured by Step 4), the interventional distribution $P(\bm{s}|\text{do}(\pi);M)$ remains invariant among all models in $\mathscr{M}_{\langle\mathcal{G}\rangle}$, i.e., $P(\bm{s}|\text{do}(\pi))$ is identifiable w.r.t. $\langle\mathcal{G},\Pi\rangle$. We could thus express $P(\bm{s}|\text{do}(\pi);M)$ for any $M\in\mathscr{M}_{\langle\mathcal{G},P\rangle}$ as a function of the observational distribution $P(\bm{o})$; for simplicity, we write $P(\bm{s}|\text{do}(\pi))=P(\bm{s}|\text{do}(\pi);M)$. The imitating policy $\pi$ is obtainable by solving the equation $P(\bm{s}|\text{do}(\pi))=P(\bm{s})$. We could derive a closed-form formula for $P(\bm{s}|\text{do}(\pi))$ following standard causal identification algorithms in . As an example, consider again the setting of Figure 1(c) with binary $X,W,S,Z$; parameters of $P(x,w,s,z)$ could be summarized using an $8$-entry probability table. The imitating policy $\pi(x)$ is thus a solution of a series of linear equations $\sum_{x}\pi(x)P(s|\text{do}(x))=P(s)$ and $\sum_{x}\pi(x)=1$, given by:

Among quantities in the above equation, $x_{i},s_{j}$ represent assignments $X=i,S=j$ for $i,j\in\{0,1\}$. The interventional distribution $P(s|\text{do}(x))$ could be identified from $P(x,w,s,z)$ using the front-door adjustment formula $P(s|\text{do}(x))=\sum_{w}P(w|x)\sum_{x^{\prime}}P(s|x^{\prime},w)P(x^{\prime})$ [29, Thm. 3.3.4].

Given a causal diagram $\mathcal{G}$, a policy space $\Pi$, and an observational distribution $P(\bm{o})$, let $\langle\bm{S},\Pi^{\prime}\rangle$ be an instrument w.r.t. $\langle\mathcal{G},\Pi\rangle$. If there exists a POSCM $M\in\mathscr{M}_{\langle\mathcal{G},P\rangle}$ and a policy $\pi\in\Pi^{\prime}$ such that $P(\bm{s}|\text{do}(\pi);M)=P(\bm{s})$, then $P(y)$ is p-imitable w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$. Moreover, $\pi$ is an imitating policy for $P(y)$ w.r.t. $\langle\mathcal{G},\Pi,P(\bm{o})\rangle$.

Experiments

We demonstrate our algorithms on several synthetic datasets, including highD consisting of natural trajectories of human driven vehicles, and on MNIST digits. In all experiments, we test our causal imitation method (ci): we apply Theorem 2 when there exists an $\pi$-backdoor admissible set; otherwise, Algorithm 1 is used to leverage the observational distribution. As a baseline, we also include naïve behavior cloning (bc) that mimics the observed conditional distribution $P(x|\mathit{pa}(\Pi))$, as well as the actual reward distribution generated by an expert (opt). We found that our algorithms consistently imitate distributions over the expert’s reward in imitable (p-imitable) cases; and p-imitable instances commonly exist. We refer readers to [15, Appendix D] for more experiments, details, and analysis.

We consider a modified example of the drone recordings of human-driven cars in Section 1 where the driver’s braking action $W$ of the left-side car is also observed. Figure 4(a) shows the causal diagram of this environment; $Z$ represent the velocity of the front-car; action $X$ represents the velocity of the driving car; $W$ and the reward signal $Y$ are both affected by an unobserved confounder $U$, representing the weather condition. In Figure 4(a), $\{Z\}$ is $\pi$-backdoor admissible while $\{Z,W\}$ is not due to active path $X\leftarrow L\rightarrow W\leftrightarrow Y$. We obtain policies for the causal and naive imitators training two separate GANs. Distributions $P(y|\text{do}(\pi))$ induced by all algorithms are reported in Figure 4(b). We also measure the L1 distance between $P(y|\text{do}(\pi))$ and the expert’s reward $P(y)$. We find that the causal approach (ci), using input set $\{Z\}$, successfully imitates $P(y)$ (L1 $=0.0018$). As expected, the naive approach (bc) utilizing all covariates $\{Z,W\}$ is unable to imitate the expert (L1 = $0.2937$).

We consider an instance of Figure 1(c) where $X,S,Y$ are binary variables; binary values of $W$ are replaced with corresponding images of MNIST digits (pictures of 1 or 0), determined based on the action $X$. For the causal imitator (ci), we learn a POSCM $\hat{M}$ such that $P(x,w,s;\hat{M})=P(x,w,s)$. To obtain $\hat{M}$, we train a GAN to imitate the observational distribution $P(x,w,s)$, with a separate generator for each $X,W,S$. We then train a separate discriminator measuring the distance between observed trajectories $P(s)$ and interventional distribution $P(s|\text{do}(\pi);\hat{M})$ over the surrogate $\{S\}$. The imitating policy is obtained by minimizing such a distance. Distributions $P(y|\text{do}(\pi))$ induced by all algorithms are reported in Figure 4(c). We find that the causal approach (ci) successfully imitates $P(y)$ (L1 $=0.0634$). As expected, the naive approach (bc) mimicking distribution $P(x)$ is unable to imitate the expert (L1 = $0.1900$).

Conclusion

We investigate the imitation learning in the semantics of structural causal models. The goal is to find an imitating policy that mimics the expert behaviors from combinations of demonstration data and qualitative knowledge about the data-generating process represented as a causal diagram. We provide a graphical criterion that is complete (i.e., sufficient and necessary) for determining the feasibility of learning an imitating policy that mimics the expert’s performance. We also study a data-dependent notion of imitability depending on the observational distribution. An efficient algorithm is introduced which finds an imitating policy, by exploiting quantitative knowledge contained in the observational data and the presence of surrogate endpoints. Finally, we propose a practical procedure for estimating such an imitating policy from observed trajectories of the expert’s demonstrations.

Broader Impact

This paper investigates the theoretical framework of learning a policy that imitates the distribution over a primary outcome from natural trajectories of an expert demonstrator, even when the primary outcome itself is unobserved and input covariates used by the expert determining original values of the action are unknown. Since in practice, the actual reward is often unspecified and the learner and the demonstrator rarely observe the environment in the same fashion, our methods are likely to increase the progress of automated decision systems. Such systems may be applicable to various fields, including the development of autonomous vehicle, industrial automation and the management of chronic disease. These applications may have a broad spectrum of societal implications. The adoption of autonomous driving and industrial automation systems could save cost and reduce risks such as occupational injuries; while it could also create unemployment. Treatment recommendation in the clinical decision support system could certainly alleviate the stress on the healthcare workers. However, this also raise questions concerning with the accountability in case of medical malpractice; collection of private personal information could also make the hospital database valuable targets for malicious hackers. Overall, we would encourage research to understand the risks arising from automated decision systems and mitigations for its negative impact.

Recently, there is a growing amount of dataset of natural vehicle trajectories like highD being licensed for commercial use. An immediate positive impact of this work is that we discuss potential risk of training decision-making policy from the observational data due to the presence of unobserved confounding, as shown in Sections 1 and 4. More broadly, since our method is based on the semantics of structural causal models [29, Ch. 7], its adoption could cultivate machine learning practitioners with proper training in causal reasoning. A favorable characteristic of causal inference methods is that they are inherently robust: for example, the definition of imitability Definition 3 requires the imitating policy to perfectly mimics the expert performance in any model compatible to the causal diagram. Automated decision systems using the causal inference methods prioritize the safety and robustness in decision-making, which is increasingly essential since the use of black-box AI systems is prevalent and our understandings of their potential implications are still limited.

Acknowledge

The authors were partially supported by grants from NSF IIS-1704352 and IIS-1750807 (CAREER).

References

Appendix A Proofs

Let $Y\in\bm{Y}\cap\bm{L}$. For any SCM $M_{1}$ that induces $\mathcal{G}$, we could obtain an SCM $M_{2}$ by replacing $f_{Y}$ and $P(u_{Y})$ associated with $Y$. Since there is no restriction on the parametrization of $f_{Y}$ and $P(u_{Y})$, we could always ensure $P(y|\text{do}(\pi);M_{1})\neq P(y|\text{do}(\pi);M_{2})$. For example, in both $M_{1},M_{2}$, let $Y\leftarrow U_{Y}$; we define $P(U_{Y}=0;M_{1})=0.1$ and $P(U_{Y}=0;M_{2})=0.9$. It is immediate to see that $P(\bm{y}|\text{do}(\pi);M_{1})\neq P(\bm{y}|\text{do}(\pi);M_{2})$, i.e., $P(\bm{y}|\text{do}(\pi))$ is not identifiable. ∎

The lack of existence of a shared imitating policy between $M_{1},M_{2}$ eliminates the possibility of the existence of a function from $P(\bm{o})$ to a policy $\pi$ that imitates $P(\bm{y})$ in any POSCM compatible with the causal diagram $\mathcal{G}$. ∎

Since there is not bi-directed arrow pointing into $X$, it is variables that $\mathit{pa}(X)_{\mathcal{G}}$ is $\pi$-backdoor admissible relative to $\langle X,Y\rangle$ in $\mathcal{G}$. By Rule 2 of do-calculus, we have

Since $\mathit{pa}(X)_{\mathcal{G}}\subseteq\mathit{Pa}(\Pi)$, the conditional distribution $P(x|\mathit{pa}(X)_{\mathcal{G}})$ can be represented as a policy in $\Pi$. Let $\pi(x|\mathit{Pa}(\Pi))=\pi(x|\mathit{pa}(X)_{\mathcal{G}})=P(x|\mathit{pa}(X)_{\mathcal{G}})$. We must have

Given a causal diagram $\mathcal{G}$ and a policy space $\Pi$, let $\bm{Z}=\mathit{An}(Y)_{\mathcal{G}}\cap\mathit{Pa}(\Pi)$. If $P(y)$ is imitable relative to $\langle\mathcal{G},\Pi\rangle$, then $\left(Y\perp\!\!\!\!\perp X|\bm{Z}\right)_{\mathcal{G}_{\underline{X}}}$.

We consider first a simplified causal diagram $\mathcal{H}$ where all endogenous variables are observed, i.e., $\bm{V}=\bm{O}$; and $\mathit{Pa}(\Pi)=\bm{O}\setminus\{X,Y\}$. In this diagram $\mathcal{G}$, $\left(Y\not\perp\!\!\!\!\perp X|\bm{Z}\right)_{\mathcal{G}_{\underline{X}}}$ implies that there exists a bi-directed path $l$ of the form $X\leftrightarrow Z_{1}\leftrightarrow Z_{2}\leftrightarrow\cdots\leftrightarrow Z_{n}\leftrightarrow Y$ such that $Z_{i}\in\bm{Z}$ for any $i=1,\dots,n$. We denote by $\bm{Z}^{*}=\{Z_{1},\dots,Z_{n}\}$. Recall that $X\in\mathit{An}(Y)_{\mathcal{H}}$. We could thus obtain a subgraph $\mathcal{H}^{\prime}$ of $\mathcal{H}$ that satisfies the following condition:

$\mathcal{H}^{\prime}$ contains the bi-directed path $l$.

All nodes in $\mathcal{H}^{\prime}$ are descendants of $\bm{Z}^{*}\cup\{X\}$.

$Y$ is a descendant for all nodes in $\mathcal{H}^{\prime}$.

Every endogenous node $V$ in $\mathcal{H}^{\prime}$ has at most one child.

All bi-directed arrows in $\mathcal{H}^{\prime}$ are contained in $l$.

A mental image for depicting $\mathcal{H}^{\prime}$ is to think of a tree rooted in node $Y$; $X,Z_{1},\dots,Z_{n}$ are leaf nodes; nodes $X,Z_{1},\dots,Z_{2},Y$ are connected by the bi-directed path $l$; $\mathcal{H}^{\prime}$ contain no bi-directed arrow except path $l$. We now construct a POSCM $M^{\prime}$ compatible with $\mathcal{G}^{\prime}$. More specifically, values of each exogenous confounder $U_{i}$ residing on $l$ are drawn uniformly over a binary domain $\{0,1\}$. For each endogenous variable $V_{i}$ in $\mathcal{H}^{\prime}$, its values is equal to the parity sum of its parents in $\mathcal{H}^{\prime}$, i.e., $V_{i}\leftarrow\oplus_{V_{j}\in\mathit{pa}(V_{i})_{\mathcal{H}^{\prime}}}V_{j}$. By construction, in the subgraph $\mathcal{H}^{\prime}$, each exogenous $U_{i}$ has exactly two directed paths going to $Y$. This means that in the constructed model $M^{\prime}$, observed values of $Y$ is always equal to $0$, i.e., the reward distribution $P(Y=0;M^{\prime})=1$.

Consider a policy space $\Pi^{\prime}$ of the form $\{\pi:\mathscr{D}_{\bm{Z}^{*}}\mapsto\mathscr{P}_{X}\}$. Let $U_{n}$ denote the exogenous confounder residing on $l$ that is closest to $Y$, i.e., $X\leftrightarrow Z_{1}\leftrightarrow Z_{2}\leftrightarrow\cdots\leftrightarrow Z_{n}\leftarrow U_{n}\rightarrow Y$. By the definition of $M^{\prime}$, values of each variable $Z_{i}\in\bm{Z}^{*}$ is the parity sum of at least two exogenous variables $U_{j},U_{k}$. Since values of each $U_{i}$ are drawn uniformly over $\{0,1\}$, we must have $P(u_{n},\bm{z}^{*})=P(u_{n})P(\bm{z}^{*})$, or equivalently, $P(u_{n}|\bm{z}^{*})=P(u_{n})$. By definition, we could obtain from $l$ a directed path going from $U_{n}$ to $Y$ that is not intercepted by $\bm{Z}^{*}$. That is, given any value $\bm{Z}^{*}=\bm{z}^{*}$, values of $Y$ is decided by a parity function taking $U_{n}$ as an input. Since $P(u_{n}|\bm{z}^{*})=P(u_{n})$ and $U_{n}$ is drawn uniformly over $\{0,1\}$, it is verifiable that in $M^{\prime}$, given any $\bm{Z}^{*}=\bm{z}^{*}$, the conditional distribution $P(Y=0|\text{do}(x),\bm{z}^{*};M^{\prime})=0.5$. This means that for any policy $\pi\in\Pi^{\prime}$, the interventional distribution $P(Y=0|\text{do}(\pi);M^{\prime})=0.5$, which is far from the observational distribution $P(Y=0;M^{\prime})=1$. That is, $P(y)$ is not imitable w.r.t. $\langle\mathcal{H}^{\prime},\Pi^{\prime}\rangle$.

We will next show the non-imitability of $P(y)$ w.r.t. $\langle\mathcal{H},\Pi\rangle$. For any node $V$ that is not included in $\mathcal{H}^{\prime}$, let its values be decided by an independent noise $U_{V}$ drawn uniformly over $\{0,1\}$. We denote this extended POSCM by $M$. Obviously, $M$ is compatible with the causal diagram $\mathcal{H}$. In this model, for any covariate $V\in\mathit{Pa}(\Pi)\setminus\bm{Z}^{*}$, it is either (1) a random variable disconnected to any other endogenous variable in $M$; or (2) decided by a function taking $\bm{Z}^{*}$ as input. That is, $V\in\mathit{Pa}(\Pi)\setminus\bm{Z}^{*}$ contains no value of information with regard to the reward $Y$ when intervening on action $X$. By [17, Ch. 23.6], this means that for any policy $\pi\in\Pi$, there exists a policy $\pi^{\prime}$ in the subspace $\Pi^{\prime}$ such that $P(y|\text{do}(\pi);M)=P(y|\text{do}(\pi^{\prime});M)$. Recall that there exists no policy $\pi$ in $\Pi^{\prime}$ that could ensure $P(y|\text{do}(\pi);M^{\prime})=P(y;M^{\prime})$ in POSCM $M^{\prime}$. By definition of $M$ and $M^{\prime}$, we must have $P(y;M)=P(y;M^{\prime})$ and for any policy $\pi\in\Pi$, $P(y|\text{do}(\pi);M)=P(y|\text{do}(\pi);M^{\prime})$. It is immediate to see that there exists no policy $\pi\in\Pi$ that could induce $P(y|\text{do}(\pi);M)=P(y;M)$ in the extended POSCM $M$, i.e., $P(y)$ is not imitable w.r.t. $\langle\mathcal{H},\Pi\rangle$.