Safe and Nested Subgame Solving for Imperfect-Information Games

Introduction

Imperfect-information games model strategic settings that have hidden information. They have a myriad of applications including negotiation, auctions, cybersecurity, and physical security.

In perfect-information games, determining the optimal strategy at a decision point only requires knowledge of the game tree’s current node and the remaining game tree beyond that node (the subgame rooted at that node). This fact has been leveraged by nearly every AI for perfect-information games, including AIs that defeated top humans in chess and Go . In checkers, the ability to decompose the game into smaller independent subgames was even used to solve the entire game . However, it is not possible to determine a subgame’s optimal strategy in an imperfect-information game using only knowledge of that subgame, because the game tree’s exact node is typically unknown. Instead, the optimal strategy may depend on the value an opponent could have received in some other, unreached subgame. Although this is counter-intuitive, we provide a demonstration in Section 2.

Rather than rely on subgame decomposition, past approaches for imperfect-information games typically solved the game as a whole upfront. For example, heads-up limit Texas hold’em, a relatively simple form of poker with $10^{13}$ decision points, was essentially solved without decomposition . However, this approach cannot extend to larger games, such as heads-up no-limit Texas hold’em—the primary benchmark in imperfect-information game solving—which has $10^{161}$ decision points .

The standard approach to computing strategies in such large games is to first generate an abstraction of the game, which is a smaller version of the game that retains as much as possible the strategic characteristics of the original game . For example, a continuous action space might be discretized. This abstract game is solved and its solution is used when playing the full game by mapping states in the full game to states in the abstract game. We refer to the solution of an abstraction (or more generally any approximate solution to a game) as a blueprint strategy.

In heavily abstracted games, a blueprint may be far from the true solution. Subgame solving attempts to improve upon the blueprint by solving in real time a more fine-grained abstraction for an encountered subgame, while fitting its solution within the overarching blueprint.

Coin Toss

In this section we provide intuition for why an imperfect-information subgame cannot be solved in isolation. We demonstrate this in a simple game we call Coin Toss, shown in Figure 1a, which will be used as a running example throughout the paper.

Coin Toss is played between players $P_{1}$ and $P_{2}$. The figure shows rewards only for $P_{1}$; $P_{2}$ always receives the negation of $P_{1}$’s reward. A coin is flipped and lands either Heads or Tails with equal probability, but only $P_{1}$ sees the outcome. $P_{1}$ then chooses between actions “Sell” and “Play.” The Sell action leads to a subgame whose details are not important, but the expected value (EV) of choosing the Sell action will be important. (For simplicity, one can equivalently assume in this section that Sell leads to an immediate terminal reward, where the value depends on whether the coin landed Heads or Tails). If the coin lands Heads, it is considered lucky and $P_{1}$ receives an EV of \0.50$for choosing Sell. On the other hand, if the coin lands Tails, it is considered unlucky and$P_{1}$receives an EV of$-\$0.50$ for action Sell. (That is, $P_{1}$ must on average pay \0.50$to get rid of the coin). If$P_{1}$instead chooses Play, then$P_{2}$may guess how the coin landed. If$P_{2}$guesses correctly, then$P_{1}$receives a reward of$-\$1$. If $P_{2}$ guesses incorrectly, then $P_{1}$ receives \1$.$P_{2}$may also forfeit, which should never be chosen but will be relevant in later sections. We wish to determine the optimal strategy for$P_{2}$in the subgame$S$that occurs after$P_{1}$ chooses Play, shown in Figure 1a.

Were $P_{2}$ to always guess Heads, $P_{1}$ would receive \0.50$for choosing Sell when the coin lands Heads, and$\$1$ for Play when it lands Tails. This would result in an average of \0.75$for$P_{1}$. Alternatively, were$P_{2}$to always guess Tails,$P_{1}$would receive$\$1$ for choosing Play when the coin lands Heads, and -\0.50$for choosing Sell when it lands Tails. This would result in an average reward of$\$0.25$ for $P_{1}$. However, $P_{2}$ would do even better by guessing Heads with $25\%$ probability and Tails with $75\%$ probability. In that case, $P_{1}$ could only receive \0.50$(on average) by choosing Play when the coin lands Heads—the same value received for choosing Sell. Similarly,$P_{1}$could only receive$-\$0.50$ by choosing Play when the coin lands Tails, which is the same value received for choosing Sell. This would yield an average reward of \0$for$P_{1}$. It is easy to see that this is the best$P_{2}$can do, because$P_{1}$can average$\$0$ by always choosing Sell. Therefore, choosing Heads with $25\%$ probability and Tails with $75\%$ probability is an optimal strategy for $P_{2}$ in the “Play” subgame.

Now suppose the coin is considered lucky if it lands Tails and unlucky if it lands Heads. That is, the expected reward for selling the coin when it lands Heads is now -\0.50$and when it lands Tails is now$\$0.50$. It is easy to see that $P_{2}$’s optimal strategy for the “Play” subgame is now to guess Heads with $75\%$ probability and Tails with $25\%$ probability. This shows that a player’s optimal strategy in a subgame can depend on the strategies and outcomes in other parts of the game. Thus, one cannot solve a subgame using information about that subgame alone. This is the central challenge of imperfect-information games as opposed to perfect-information games.

Notation and Background

This paper focuses on two-player zero-sum games. In a two-player zero-sum extensive-form game there are two players, $\mathcal{P}=\{1,2\}$. $H$ is the set of all possible nodes, represented as a sequence of actions. $A(h)$ is the actions available in a node and $P(h)\in\mathcal{P}\cup c$ is the player who acts at that node, where $c$ denotes chance. Chance plays an action $a\in A(h)$ with a fixed probability. If action $a\in A(h)$ leads from $h$ to $h^{\prime}$, then we write $h\cdot a=h^{\prime}$. If a sequence of actions leads from $h$ to $h^{\prime}$, then we write $h\sqsubset h^{\prime}$. The set of nodes $Z\subseteq H$ are terminal nodes. For each player $i\in\mathcal{P}$, there is a payoff function $u_{i}:Z\rightarrow\Re$ where $u_{1}=-u_{2}$.

Imperfect information is represented by information sets (infosets). Every node $h\in H$ belongs to exactly one infoset for each player. For any infoset $I_{i}$, nodes $h,h^{\prime}\in I_{i}$ are indistinguishable to player $i$. Thus the same player must act at all the nodes in an infoset, and the same actions must be available. Let $P(I_{i})$ and $A(I_{i})$ be such that all $h\in I_{i}$, $P(I_{i})=P(h)$ and $A(I_{i})=A(h)$.

A strategy $\sigma_{i}(I_{i})$ is a probability vector over $A(I_{i})$ for infosets where $P(I_{i})=i$. The probability of action $a$ is denoted by $\sigma_{i}(I_{i},a)$. For all $h\in I_{i}$, $\sigma_{i}(h)=\sigma_{i}(I_{i})$. A full-game strategy $\sigma_{i}\in\Sigma_{i}$ defines a strategy for each player $i$ infoset. A strategy profile $\sigma$ is a tuple of strategies, one for each player. The expected payoff for player $i$ if all players play the strategy profile $\langle\sigma_{i},\sigma_{-i}\rangle$ is $u_{i}(\sigma_{i},\sigma_{-i})$, where $\sigma_{-i}$ denotes the strategies in $\sigma$ of all players other than $i$.

Let $\pi^{\sigma}(h)=\prod_{h^{\prime}\cdot a\sqsubseteq h}\sigma_{P(h^{\prime})}(h^{\prime},a)$ denote the probability of reaching $h$ if all players play according to $\sigma$. $\pi^{\sigma}_{i}(h)$ is the contribution of player $i$ to this probability (that is, the probability of reaching $h$ if chance and all players other than $i$ always chose actions leading to $h$). $\pi^{\sigma}_{-i}(h)$ is the contribution of all players, and chance, other than $i$. We similarly define $\pi^{\sigma}(h,h^{\prime})$ is the probability of reaching $h^{\prime}$ given that $h$ has been reached, and if $h\not\sqsubset h^{\prime}$. This papers focuses on perfect-recall games, where a player never forgets past information. Thus, for every $I_{i}$, $\forall h,h^{\prime}\in I_{i}$, $\pi^{\sigma}_{i}(h)=\pi^{\sigma}_{i}(h^{\prime})$. We define $\pi^{\sigma}_{i}(I_{i})=\pi^{\sigma}_{i}(h)$ for $h\in I_{i}$. Also, $I^{\prime}_{i}\sqsubset I_{i}$ if for some $h^{\prime}\in I^{\prime}_{i}$ and some $h\in I_{i}$, $h^{\prime}\sqsubset h$. Similarly, $I^{\prime}_{i}\cdot a\sqsubset I_{i}$ if $h^{\prime}\cdot a\sqsubset h$.

A Nash equilibrium is a strategy profile $\sigma^{*}$ where no player can improve by shifting to a different strategy, so $\sigma^{*}$ satisfies $\forall i,\ u_{i}(\sigma^{*}_{i},\sigma^{*}_{-i})=\max_{\sigma^{\prime}_{i}\in\Sigma_{i}}u_{i}(\sigma^{\prime}_{i},\sigma^{*}_{-i})$. An $\epsilon$-Nash equilibrium is a strategy profile $\sigma^{*}$ such that $\forall i,\ u_{i}(\sigma^{*}_{i},\sigma^{*}_{-i})+\epsilon\geq\max_{\sigma^{\prime}_{i}\in\Sigma_{i}}u_{i}(\sigma^{\prime}_{i},\sigma^{*}_{-i})$. A best response $BR(\sigma_{-i})$ is a strategy for player $i$ that is optimal against $\sigma_{-i}$. Formally, $BR(\sigma_{-i})$ satisfies $u_{i}(BR(\sigma_{-i}),\sigma_{-i})=\max_{\sigma^{\prime}_{i}\in\Sigma_{i}}u_{i}(\sigma_{i}^{\prime},\sigma_{-i})$. In a two-player zero-sum game, the exploitability $\textit{exp}(\sigma_{i})$ of a strategy $\sigma_{i}$ is how much worse $\sigma_{i}$ does against an opponent best response than a Nash equilibrium strategy would do. Formally, exploitability of $\sigma_{i}$ is $u_{i}(\sigma^{*})-u_{i}(\sigma_{i},BR(\sigma_{i}))$, where $\sigma^{*}$ is a Nash equilibrium.

The expected value of a node $h$ when players play according to $\sigma$ is v_{i}^{\sigma}(h)=\sum_{z\in Z}\big{(}\pi^{\sigma}(h,z)u_{i}(z)\big{)}. An infoset’s value is the weighted average of the values of the nodes in the infoset, where a node is weighed by the player’s belief that she is in that node. Formally, v_{i}^{\sigma}(I_{i})=\frac{\sum_{h\in I_{i}}\big{(}\pi_{-i}^{\sigma}(h)v_{i}^{\sigma}(h)\big{)}}{\sum_{h\in I_{i}}\pi_{-i}^{\sigma}(h)} and v_{i}^{\sigma}(I_{i},a)=\frac{\sum_{h\in I_{i}}\big{(}\pi_{-i}^{\sigma}(h)v_{i}^{\sigma}(h\cdot a)\big{)}}{\sum_{h\in I_{i}}\pi_{-i}^{\sigma}(h)}. A counterfactual best response $CBR(\sigma_{-i})$ is a best response that also maximizes value in unreached infosets. Specifically, a counterfactual best response is a best response $\sigma_{i}$ with the additional condition that if $\sigma_{i}(I_{i},a)>0$ then $v_{i}^{\sigma}(I_{i},a)=\max_{a^{\prime}}v_{i}^{\sigma}(I_{i},a^{\prime})$. We further define counterfactual best response value $CBV^{\sigma_{-i}}(I_{i})$ as the value player $i$ expects to achieve by playing according to $CBR(\sigma_{-i})$, having already reached infoset $I_{i}$. Formally, $CBV^{\sigma_{-i}}(I_{i})=v_{i}^{\langle CBR({\sigma_{-i}}),\sigma_{-i}\rangle}(I_{i})$ and $CBV^{\sigma_{-i}}(I_{i},a)=v_{i}^{\langle CBR({\sigma_{-i}}),\sigma_{-i}\rangle}(I_{i},a)$.

An imperfect-information subgame, which we refer to simply as a subgame in this paper, can in most cases (but not all) be described as including all nodes which share prior public actions (that is, actions viewable to both players). In poker, for example, a subgame is uniquely defined by a sequence of bets and public board cards. Figure 1b shows the public game tree of Coin Toss. Formally, an imperfect-information subgame is a set of nodes $S\subseteq H$ such that for all $h\in S$, if $h\sqsubset h^{\prime}$, then $h^{\prime}\in S$, and for all $h\in S$ and all $i\in\mathcal{P}$, if $h^{\prime}\in I_{i}(h)$ then $h^{\prime}\in S$. Define $S_{\textit{top}}$ as the set of earliest-reachable nodes in $S$. That is, $h\in S_{\textit{top}}$ if $h\in S$ and $h^{\prime}\not\in S$ for any $h^{\prime}\sqsubset h$.

Prior Approaches to Subgame Solving

This section reviews prior techniques for subgame solving in imperfect-information games, which we build upon. Throughout this section, we refer to the Coin Toss game shown in Figure 1a.

As discussed in Section 1, a standard approach to dealing with large imperfect-information games is to solve an abstraction of the game. The abstract solution is a (probably suboptimal) strategy profile in the full game. We refer to this full-game strategy profile as the blueprint. The goal of subgame solving is to improve upon the blueprint by changing the strategy only in a subgame. While the blueprint is frequently a Nash equilibrium (or approximate Nash equilibrium) in some abstraction of the full game, our techniques do not assume this. The blueprint can in fact be any arbitrary strategy in the full game.

Assume that a blueprint $\sigma$ (shown in Figure 2) has already been computed for Coin Toss in which $P_{1}$ chooses Play $\frac{3}{4}$ of the time with Heads and $\frac{1}{2}$ of the time with Tails, and $P_{2}$ chooses Heads $\frac{1}{2}$ of the time, Tails $\frac{1}{4}$ of the time, and Forfeit $\frac{1}{4}$ of the time after $P_{1}$ chooses Play. In many large games the blueprint is far from optimal either because the equilibrium-finding algorithm did not sufficiently converge or because the game was too large and had to be abstracted. Clearly the example blueprint shown here could be trivially improved; we use it for simplicity of exposition. The details of the blueprint in the Sell subgame are not relevant in this section, but the EV for choosing the Sell action is relevant. We assume that if $P_{1}$ chose the Sell action and played optimally thereafter, then she would receive an expected payoff of $0.5$ if the coin is Heads, and $-0.5$ if the coin is Tails. We will attempt to improve $P_{2}$’s strategy in the subgame $S$ that follows $P_{1}$ choosing Play.

We first review the most intuitive form of subgame solving, which we refer to as Unsafe subgame solving . This form of subgame solving assumes both players played according to the blueprint prior to reaching the subgame. That defines a probability distribution over the nodes at the root of the subgame $S$, representing the probability that the true game state matches that node. A strategy for the subgame is then calculated which assumes that this distribution is correct.

In all subgame solving algorithms, an augmented subgame containing $S$ and a few additional nodes is solved to determine the strategy for $S$. Applying Unsafe subgame solving to the blueprint in Coin Toss (after $P_{1}$ chooses Play) means solving the augmented subgame shown in Figure 3(a).

Specifically, the augmented subgame consists of only an initial chance node and $S$. The initial chance node reaches $h\in S_{\textit{top}}$ with probability $\frac{\pi^{\sigma}(h)}{\sum_{h^{\prime}\in S_{\textit{top}}}\pi^{\sigma}(h^{\prime})}$. The augmented subgame is solved and its strategy for $P_{2}$ is used in $S$ rather than the blueprint strategy.

Unsafe subgame solving lacks theoretical solution quality guarantees and there are many situations where it performs extremely poorly. Indeed, if it were applied to the blueprint of Coin Toss then $P_{2}$ would always choose Heads—which $P_{1}$ could exploit severely by only choosing Play with Tails. Despite the lack of theoretical guarantees and potentially bad performance, Unsafe subgame solving is simple and can sometimes produce low-exploitability strategies, as we show later.

We now move to discussing safe subgame-solving techniques, that is, ones that ensure that the exploitability of the strategy is no higher than that of the blueprint strategy.

2 Subgame Resolving

In subgame Resolving , a safe strategy is computed for $P_{2}$ in the subgame by solving the augmented subgame shown in Figure 3(b), producing an equilibrium strategy $\sigma^{S}$. This augmented subgame differs from Unsafe subgame solving by giving $P_{1}$ the option to “opt out” from entering $S$ and instead receive the EV of playing optimally against $P_{2}$’s blueprint strategy in $S$.

Specifically, the augmented subgame for Resolving differs from unsafe subgame solving as follows. For each $h_{\textit{top}}\in S_{\textit{top}}$ we insert a new $P_{1}$ node $h_{r}$, which exists only in the augmented subgame, between the initial chance node and $h_{\textit{top}}$. The set of these $h_{r}$ nodes is $S_{r}$. The initial chance node connects to each node $h_{r}\in S_{r}$ in proportion to the probability that player $P_{1}$ could reach $h_{\textit{top}}$ if $P_{1}$ tried to do so (that is, in proportion to $\pi_{-1}^{\sigma}(h_{\textit{top}})$). At each node $h_{r}\in S_{r}$, $P_{1}$ has two possible actions. Action $a^{\prime}_{S}$ leads to $h_{\textit{top}}$, while action $a^{\prime}_{T}$ leads to a terminal payoff that awards the value of playing optimally against $P_{2}$’s blueprint strategy, which is $CBV^{\sigma_{2}}(I_{1}(h_{\textit{top}}))$. In the blueprint of Coin Toss, $P_{1}$ choosing Play after the coin lands Heads results in an EV of , and $\frac{1}{2}$ if the coin is Tails. Therefore, $a^{\prime}_{T}$ leads to a terminal payoff of for Heads and $\frac{1}{2}$ for Tails. After the equilibrium strategy $\sigma^{S}$ is computed in the augmented subgame, $P_{2}$ plays according to the computed subgame strategy $\sigma^{S}_{2}$ rather than the blueprint strategy when in $S$. The $P_{1}$ strategy $\sigma^{S}_{1}$ is not used.

Clearly $P_{1}$ cannot do worse than always picking action $a^{\prime}_{T}$ (which awards the highest EV $P_{1}$ could achieve against $P_{2}$’s blueprint). But $P_{1}$ also cannot do better than always picking $a^{\prime}_{T}$, because $P_{2}$ could simply play according to the blueprint in $S$, which means action $a^{\prime}_{S}$ would give the same EV to $P_{1}$ as action $a^{\prime}_{T}$ (if $P_{1}$ played optimally in $S$). In this way, the strategy for $P_{2}$ in $S$ is pressured to be no worse than that of the blueprint. In Coin Toss, if $P_{2}$ were to always choose Heads (as was the case in Unsafe subgame solving), then $P_{1}$ would always choose $a^{\prime}_{T}$ with Heads and $a^{\prime}_{S}$ with Tails.

Resolving guarantees that $P_{2}$’s exploitability will be no higher than the blueprint’s (and may be better). However, it may miss opportunities for improvement. For example, if we apply Resolving to the example blueprint in Coin Toss, one solution to the augmented subgame is the blueprint itself, so $P_{2}$ may choose Forfeit $25\%$ of the time even though Heads and Tails dominate that action. Indeed, the original purpose of Resolving was not to improve upon a blueprint strategy in a subgame, but rather to compactly store it by keeping only the EV at the root of the subgame and then reconstructing the strategy in real time when needed rather than storing the whole subgame strategy.

Maxmargin subgame solving , discussed in Appendix A, can improve performance by defining a margin $M^{\sigma^{S}}(I_{1})=CBV^{\sigma_{2}}(I_{1})-CBV^{\sigma^{S}_{2}}(I_{1})$ for each $I_{1}\in S_{\textit{top}}$ and maximizing $\min_{I_{1}\in S_{\textit{top}}}M^{\sigma^{S}}(I_{1})$. Resolving only makes all margins nonnegative. However, Maxmargin does worse in practice when using estimates of equilibrium values as discussed in Section 6.

Reach Subgame Solving

All of the subgame-solving techniques described in Section 4 only consider the target subgame in isolation, which can lead to suboptimal strategies. For example, Maxmargin solving applied to $S$ in Coin Toss results in $P_{2}$ choosing Heads with probability $\frac{5}{8}$ and Tails with $\frac{3}{8}$ in $S$. This results in $P_{1}$ receiving an EV of $-\frac{1}{4}$ by choosing Play in the Heads state, and an EV of $\frac{1}{4}$ in the Tails state. However, $P_{1}$ could simply always choose Sell in the Heads state (earning an EV of $0.5$) and Play in the Tails state and receive an EV of $\frac{3}{8}$ for the entire game. In this section we introduce Reach subgame solving, an improvement to past subgame-solving techniques that considers what the opponent could have alternatively received from other subgames.Other subgame-solving methods have also considered the cost of reaching a subgame . However, those approaches are not correct in theory when applied in real time to any subgame reached during play. For example, a better strategy for $P_{2}$ would be to choose Heads with probability $\frac{3}{4}$ and Tails with probability $\frac{1}{4}$. Then $P_{1}$ is indifferent between choosing Sell and Play in both cases and overall receives an expected payoff of for the whole game.

However, that strategy is only optimal if $P_{1}$ would indeed achieve an EV of $0.5$ for choosing Sell in the Heads state and $-0.5$ in the Tails state. That would be the case if $P_{2}$ played according to the blueprint in the Sell subgame (which is not shown), but in reality we would apply subgame solving to the Sell subgame if the Sell action were taken, which would change $P_{2}$’s strategy there and therefore $P_{1}$’s EVs. Applying subgame solving to any subgame encountered during play is equivalent to applying it to all subgames independently. Thus, we must consider that the EVs from other subgames may differ from what the blueprint says because subgame solving would be applied to them as well.

As an example of this issue, consider the game shown in Figure 4 which contains two identical subgames $S_{1}$ and $S_{2}$ where the blueprint has $P_{2}$ pick Heads and Tails with 50% probability. The Sell action leads to an EV of $0.5$ from the Heads state, while Play leads to an EV of . If we were to solve just $S_{1}$, then $P_{2}$ could afford to always choose Tails in $S_{1}$, thereby letting $P_{1}$ achieve an EV of $1$ for reaching that subgame from Heads because, due to the chance node $C_{1}$, $S_{1}$ is only reached with 50% probability. Thus, $P_{1}$’s EV for choosing Play would be $0.5$ from Heads and $-0.5$ from Tails, which is optimal. We can achieve this strategy in $S_{1}$ by solving an augmented subgame in which the alternative payoff for Heads is $1$. In that augmented subgame, $P_{2}$ always choosing Tails would be a solution (though not the only solution).

However, if the same reasoning were applied independently to $S_{2}$ as well, then $P_{2}$ might always choose Tails in both subgames and $P_{1}$’s EV for choosing Play from Heads would become $1$ while the EV for Sell would only be $0.5$. Instead, we could allow $P_{1}$ to achieve an EV of $0.5$ for reaching each subgame from Heads (by setting the alternative payoff for Heads to $0.5$). In that case, $P_{1}$’s overall EV for choosing Play could only increase to $0.5$, even if both $S_{1}$ and $S_{2}$ were solved independently.

We capture this intuition by considering for each $I_{1}\in S_{\textit{top}}$ all the infosets and actions $I^{\prime}_{1}\cdot a^{\prime}\sqsubset I_{1}$ that $P_{1}$ would have taken along the path to $I_{1}$. If, at some $I^{\prime}_{1}\cdot a^{\prime}\sqsubset I_{1}$ where $P_{1}$ acted, there was a different action $a^{*}\in A(I^{\prime}_{1})$ that leads to a higher EV, then $P_{1}$ would have taken a suboptimal action if they reached $I_{1}$. The difference in value between $a^{*}$ and $a^{\prime}$ is referred to as a gift. We can afford to let $P_{1}$’s value for $I_{1}$ increase beyond the blueprint value (and in the process lower $P_{1}$’s value in some other infoset in $S_{\textit{top}}$), so long as the increase to $I_{1}$’s value is small enough that choosing actions leading to $I_{1}$ is still suboptimal for $P_{1}$. Critically, we must ensure that the increase in value is small enough even when the potential increase across all subgames is summed together, as in Figure 4.In this paper and in our experiments, we allow any infoset that descends from a gift to increase by the size of the gift (e.g., in Figure 4 the gift from Heads is $0.5$, so we allow $P_{1}$’s value for Heads in both $S_{1}$ and $S_{2}$ to increase by $0.5$). However, any division of the gift among subgames is acceptable so long as the potential increase across all subgames (multiplied by the probability of $P_{1}$ reaching that subgame) does not exceed the original gift. For example in Figure 4 if we only apply Reach subgame solving to $S_{1}$, then we could allow the Heads state in $S_{1}$ to increase by $1$ rather than just by $0.5$. In practice, some divisions may do better than others. The division we use in this paper (applying gifts equally to all subgames) did well in practice.

A complicating factor is that gifts we assumed were present may actually not exist. For example, in Coin Toss, suppose applying subgame solving to the Sell subgame results in $P_{1}$’s value for Sell from the Heads state decreasing from $0.5$ to $0.25$. If we independently solve the Play subgame, we have no way of knowing that $P_{1}$’s value for Sell is lower than the blueprint suggested, so we may still assume there is a gift of $0.5$ from the Heads state based on the blueprint. Thus, in order to guarantee a theoretical result on exploitability that is as strong as possible, we use in our theory and experiments a lower bound on what gifts could be after subgame solving was applied to all other subgames.

Formally, let $\sigma_{2}$ be a $P_{2}$ blueprint and let $\sigma^{-S}_{2}$ be the $P_{2}$ strategy that results from applying subgame solving independently to a set of disjoint subgames other than $S$. Since we do not want to compute $\sigma_{2}^{-S}$ in order to apply subgame solving to $S$, let $\lfloor g^{\sigma_{2}^{-S}}(I^{\prime}_{1},a^{\prime})\rfloor$ be a lower bound of $CBV^{\sigma^{-S}_{2}}(I^{\prime}_{1})-CBV^{\sigma^{-S}_{2}}(I^{\prime}_{1},a^{\prime})$ that does not require knowledge of $\sigma_{2}^{-S}$. In our experiments we use $\lfloor g^{\sigma_{2}^{-S}}(I^{\prime}_{1},a^{\prime})\rfloor=\max_{a\in A_{z}(I^{\prime}_{1})\cup\{a^{\prime}\}}CBV^{\sigma_{2}}(I^{\prime}_{1},a)-CBV^{\sigma_{2}}(I^{\prime}_{1},a^{\prime})$ where $A_{z}(I^{\prime}_{1})\subseteq A(I^{\prime}_{1})$ is the set of actions leading immediately to terminal nodes. Reach subgame solving modifies the augmented subgame in Resolving and Maxmargin by increasing the alternative payoff for infoset $I_{1}\in S_{\textit{top}}$ by $\sum_{I^{\prime}_{1}\cdot a^{\prime}\sqsubseteq I_{1}\mid P(I^{\prime}_{1})=P_{1}}\lfloor g^{\sigma_{2}^{-S}}(I^{\prime}_{1},a^{\prime})\rfloor$. Formally, we define a reach margin as

This margin is larger than or equal to the one for Maxmargin, because $\lfloor g^{\sigma_{2}^{-S}}(I^{\prime},a^{\prime})\rfloor$ is nonnegative. We refer to the improved algorithms as Reach-Resolve and Reach-Maxmargin.

Intuitively, the alternative payoff in an augmented subgame determines how important it is that $P_{2}$ “defend” against that $P_{1}$ infoset. If the alternative payoff is increased, then $P_{1}$ is more likely to choose the alternative payoff rather than enter the subgame, so $P_{2}$ can instead focus on lowering the value of other $P_{1}$ infosets in $S_{\textit{top}}$.

Theorem 1 shows that when subgames are solved independently and using lower bounds on gifts, Reach-Maxmargin solving has exploitability lower than or equal to past safe techniques. The theorem statement is similar to that of Maxmargin , but the margins are now larger (or equal) in size.

So far the described techniques have guaranteed a reduction in exploitability over the blueprint by setting the value of $a^{\prime}_{T}$ equal to the value of $P_{1}$ playing optimally to $P_{2}$’s blueprint. Relaxing this guarantee by instead setting the value of $a^{\prime}_{T}$ equal to an estimate of $P_{1}$’s value when both players play optimally leads to far lower exploitability in practice. We discuss this approach in the next section.

Estimates for Alternative Payoffs

In this section we consider the case where we have a good estimate of what the values of subgames would look like in a Nash equilibrium. Unlike previous sections, exploitability might be higher than the blueprint when using this method; the solution quality ultimately depends on the accuracy of the estimates used. In practice this approach leads to significantly lower exploitability.

When solving multiple $P_{2}$ subgames, there is a minimally-exploitable strategy $\sigma^{*}_{2}$ that could, in theory, be computed by changing only the strategies in the subgames. ($\sigma^{*}_{2}$ may not be a Nash equilibrium because $P_{2}$’s strategy outside the subgames is fixed, but it is the closest that can be achieved by changing the strategy only in the subgames). However, $\sigma^{*}_{2}$ can only be guaranteed to be produced by solving all the subgames together, because the optimal strategy in one subgame depends on the optimal strategy in other subgames.

Still, suppose that we know $CBV^{\sigma^{*}_{2}}(I_{1})$ for every infoset $I_{1}\in S_{\textit{top}}$ for every subgame $S$. Let $I_{r,1}$ be the infoset in $S_{r}$ that leads to $I_{1}$. By setting the $P_{1}$ alternative payoff for $I_{r,1}$ to $v(I_{r,1},a^{\prime}_{T})=CBV^{\sigma^{*}_{2}}(I_{1})$, safe subgame solving guarantees a strategy will be produced with exploitability no worse than $\sigma^{*}_{2}$. Thus, achieving a strategy equivalent to $\sigma^{*}_{2}$ does not require knowledge of $\sigma^{*}_{2}$; rather, it only requires knowledge of $CBV^{\sigma^{*}_{2}}(I_{1})$ for infosets $I_{1}$ in the top of the subgames.

If the blueprint was produced by conducting $T$ iterations of CFR in an abstract game, then one could instead simply use the final iteration’s strategy $\sigma_{1}^{T}$, as this converges to a counterfactual best response within the abstract game. This is what we use in our experiments in this paper.

Theorem 2 proves that if we use estimates of $CBV^{\sigma^{*}_{2}}(I_{1})$ as the alternative payoffs in Maxmargin subgame solving, then we can bound exploitability by the distance of the estimates from the true values. This is in contrast to the previous algorithms which guaranteed exploitability no worse than the blueprint.

Using estimates of the values of $\sigma^{*}$ tends to be do better than the theoretically safe options described in Section 4.It is also possible to combine the safety of past approaches with some of the better performance of using estimates by adding the original Resolve conditions as additional constraints.

Although Theorem 2 uses Maxmargin in the proof, in practice Resolve does far better with estimates than Maxmargin. Additionally, the theorem easily extends to Reach-Maxmargin as well, and Reach-Resolve does better than Resolve regardless of whether estimates are used.

Section B.1 discusses an improvement, which we refer to as Distributional alternative payoffs, that leads to even better performance by making the algorithm more robust to errors in the blueprint estimates.

Nested Subgame Solving

As we have discussed, large games must be abstracted to reduce the game to a tractable size. This is particularly common in games with large or continuous action spaces. Typically the action space is discretized by action abstraction so that only a few actions are included in the abstraction. While we might limit ourselves to the actions we included in the abstraction, an opponent might choose actions that are not in the abstraction. In that case, the off-tree action can be mapped to an action that is in the abstraction, and the strategy from that in-abstraction action can be used. For example, in an auction game we might include a bid of \100$in our abstraction. If a player bids$\$101$, we simply treat that as a bid of \100$. This is referred to as action translation . Action translation is the state-of-the-art prior approach to dealing with this issue. It has been used, for example, by all the leading competitors in the Annual Computer Poker Competition (ACPC).

In this section, we develop techniques for applying subgame solving to calculate responses to opponent off-tree actions, thereby obviating the need for action translation. That is, rather than simply treat a bid of \101$as$\$100$, we calculate in real time a unique response to the bid of \101$. This can also be done in a nested fashion in response to subsequent opponent off-tree actions. We present two methods that dramatically outperform the leading action translation technique. Additionally, these techniques can be used to solve finer-grained models as play progresses down the game tree. For exposition, we assume that$P_{2}$wishes to respond to$P_{1}$ choosing an off-tree action.

We refer to the first method as the inexpensive method.Following our study, the AI DeepStack used a technique similar to this form of nested subgame solving . When $P_{1}$ chooses an off-tree action $a$, a subgame $S$ is generated following that action such that for any infoset $I_{1}$ that $P_{1}$ might be in, $I_{1}\cdot a\in S_{\textit{top}}$. This subgame may itself be an abstraction. A solution $\sigma^{S}$ is computed via subgame solving, and $\sigma^{S}$ is combined with $\sigma$ to form a new blueprint $\sigma^{\prime}$ in the expanded abstraction that now includes action $a$. The process repeats whenever $P_{1}$ again chooses an off-tree action.

The “inexpensive” approach cannot be combined with Unsafe subgame solving because the probability of reaching an action outside of a player’s abstraction is undefined. Nevertheless, a similar approach is possible with Unsafe subgame solving (as well as all the other subgame-solving techniques) by starting the subgame solving at $h$ rather than at $h\cdot a$. In other words, if action $a$ taken in node $h$ is not in the abstraction, then Unsafe subgame solving is conducted in the smallest subgame containing $h$ (and action $a$ is added to that abstraction). This increases the size of the subgame compared to the inexpensive method because a strategy must be recomputed for every action $a^{\prime}\in A(h)$ in addition to $a$. For example, if an off-tree action is chosen by the opponent as the first action in the game, then the strategy for the entire game must be recomputed. We therefore call this method the expensive method. We present experiments with both methods.

Experiments

Our experiments were conducted on heads-up no-limit Texas hold’em, as well as two smaller-scale poker games we call No-Limit Flop Hold’em (NLFH) and No-Limit Turn Hold’em (NLTH). The description for these games can be found in Appendix E. For equilibrium finding, we used CFR+ .

Our first experiment compares the performance of the subgame-solving techniques when applied to information abstraction (which is card abstraction in the case of poker). Specifically, we solve NLFH with no information abstraction on the preflop. On the flop, there are 1,286,792 infosets for each betting sequence; the abstraction buckets them into 200, 2,000, or 30,000 abstract ones (using a leading information abstraction algorithm ). We then apply subgame solving immediately after the flop community cards are dealt.

We experiment with two versions of the game, one small and one large, which include only a few of the available actions in each infoset. We also experimented on abstractions of NLTH. In that case, we solve NLTH with no information abstraction on the preflop or flop. On the turn, there are 55,190,538 infosets for each betting sequence; the abstraction buckets them into 200, 2,000, or 20,000 abstract ones. We apply subgame solving immediately after the turn community card is dealt.

Tables 1, 2, and 3 show the performance of each technique. In all our experiments, exploitability is measured in the standard units used in this field: milli big blinds per hand (mbb/h).

In the above experiments, Estimate is the technique introduced in Section 6 (added on top of Resolving) and Distributional is the technique introduced in Appendix B.1. We use a normal distribution in the Distributional subgame solving experiments, with standard deviation determined by the heuristic presented in Appendix B.1.

Since subgame solving begins immediately after a chance node with an extremely high branching factor ($1,755$ in NLFH), the gifts for the Reach algorithms are divided among subgames inefficiently. Many subgames do not use the gifts at all, while others could make use of more. The result is that the theoretically safe version of Reach allocates gifts very conservatively. In the experiments we show results both for the theoretically safe splitting of gifts, as well as a more aggressive version where gifts are scaled up by the branching factor of the chance node ($1,755$). This weakens the theoretical guarantees of the algorithm, but in general did better than splitting gifts in a theoretically correct manner. However, this is not universally true. Appendix D shows that in at least one case, exploitability increased when gifts were scaled up too aggressively. In all cases, using Reach subgame solving in at least the theoretical safe method led to lower exploitability.

Despite lacking theoretical guarantees, Unsafe subgame solving did surprisingly well in most games. However, it did substantially worse in Large NLFH with 30,000 buckets. This exemplifies its variability. Among the safe methods, all of the changes we introduce show improvement over past techniques. The Reach-Estimate + Distributional algorithm generally resulted in the lowest exploitability among the various choices, and in most cases beat Unsafe subgame solving.

In all but one case, using estimated values lowered exploitability more than Maxmargin and Resolve subgame solving. Also, in all but one case using distributional alternative payoffs lowered exploitability.

The second experiment evaluates nested subgame solving, and compares it to action translation. In order to also evaluate action translation, in this experiment, we create an NLFH game that includes 3 bet sizes at every point in the game tree (0.5, 0.75, and 1.0 times the size of the pot); a player can also decide not to bet. Only one bet (i.e., no raises) is allowed on the preflop, and three bets are allowed on the flop. There is no information abstraction anywhere in the game. We also created a second, smaller abstraction of the game in which there is still no information abstraction, but the 0.75$\times$ pot bet is never available. We calculate the exploitability of one player using the smaller abstraction, while the other player uses the larger abstraction. Whenever the large-abstraction player chooses a 0.75$\times$ pot bet, the small-abstraction player generates and solves a subgame for the remainder of the game (which again does not include any subsequent 0.75$\times$ pot bets) using the nested subgame-solving techniques described above. This subgame strategy is then used as long as the large-abstraction player plays within the small abstraction, but if she chooses the 0.75$\times$ pot bet again later, then the subgame solving is used again, and so on.

Table 4 shows that all the subgame-solving techniques substantially outperform action translation. Resolve, Maxmargin, and Reach-Maxmargin use inexpensive nested subgame solving, while Unsafe and “Reach-Maxmargin (expensive)” use the expensive approach. In all cases, we used estimates for the alternative payoff as described in Section 7. We did not test distributional alternative payoffs in this experiment, since the calculated best response values are likely quite accurate. Reach-Maxmargin performed the best, outperforming Maxmargin and Unsafe subgame solving. These results suggest that nested subgame solving is preferable to action translation (if there is sufficient time to solve the subgame).

We used the techniques presented in this paper in our AI Libratus, which competed against four top human specialists in heads-up no-limit Texas hold’em in the January 2017 Brains vs. AI competition. Libratus was constructed by first solving an abstraction of the game via a new variant of Monte Carlo CFR that prunes negative-regret actions . Libratus applied nested subgame solving (solved with CFR+ ) upon reaching the third betting round, and in response to every subsequent opponent bet thereafter. This allowed Libratus to avoid information abstraction during play, and leverage nested subgame solving’s far lower exploitability in response to opponent off-tree actions.

No-limit Texas hold’em is the most popular form of poker in the world and has been the primary benchmark challenge for AI in imperfect-information games. The competition was played over the course of 20 days, and involved 120,000 hands of poker. A prize pool of $200,000 was split among the four humans based on their performance against the AI to incentivize strong play. The AI decisively defeated the team of human players by a margin of$147$ mbb / hand, with 99.98 statistical significance (see Figure 5). This was the first, and so far only, time an AI defeated top humans in no-limit poker.

Conclusion

We introduced a subgame-solving technique for imperfect-information games that has stronger theoretical guarantees and better practical performance than prior subgame-solving methods. We presented results on exploitability of both safe and unsafe subgame-solving techniques. We also introduced a method for nested subgame solving in response to the opponent’s off-tree actions, and demonstrated that this leads to dramatically better performance than the usual approach of action translation. This is, to our knowledge, the first time that exploitability of subgame-solving techniques has been measured in large games.

Finally, we demonstrated the effectiveness of these techniques in practice in heads-up no-limit Texas Hold’em poker, the main benchmark challenge for AI in imperfect-information games. We developed the first AI to reach the milestone of defeating top humans in heads-up no-limit Texas Hold’em.

Acknowledgments

This material is based on work supported by the National Science Foundation under grants IIS-1718457, IIS-1617590, and CCF-1733556, and the ARO under award W911NF-17-1-0082, as well as XSEDE computing resources provided by the Pittsburgh Supercomputing Center. The Brains vs. AI competition was sponsored by Carnegie Mellon University, Rivers Casino, GreatPoint Ventures, Avenue4Analytics, TNG Technology Consulting, Artificial Intelligence, Intel, and Optimized Markets, Inc. We thank Kristen Gardner, Marcelo Gutierrez, Theo Gutman-Solo, Eric Jackson, Christian Kroer, Tim Reiff, and the anonymous reviewers for helpful feedback.

References

Appendix A Maxmargin Solving

Maxmargin solving is similar to Resolving, except that it seeks to improve $P_{2}$’s strategy in the subgame strategy as much as possible. While Resolving seeks a strategy for $P_{2}$ in $S$ that would simply dissuade $P_{1}$ from entering $S$, Maxmargin solving additionally seeks to punish $P_{1}$ as much as possible if $P_{1}$ nevertheless chooses to enter $S$. A subgame margin is defined for each infoset in $S_{r}$, which represents the difference in value between entering the subgame versus choosing the alternative payoff. Specifically, for each infoset $I_{1}\in S_{\textit{top}}$, the subgame margin is

In Maxmargin solving, a Nash equilibrium $\sigma^{S}$ for the augmented subgame described in Resolving subgame solving is computed such that the minimum margin over all $I_{1}\in S_{\textit{top}}$ is maximized. Aside from maximizing the minimum margin, the augmented subgames used in Resolving and Maxmargin solving are identical.

Given our base strategy in Coin Toss, Maxmargin solving would result in $P_{2}$ choosing Heads with probability $\frac{5}{8}$, Tails with probability $\frac{3}{8}$, and Forfeit with probability .

The augmented subgame can be solved in a way that maximizes the minimum margin by using a standard LP solver. In order to use iterative algorithms such as the Excessive Gap Technique or Counterfactual Regret Minimization (CFR) , one can use the gadget game described by Moravcik et al. . Details on the gadget game are provided in the Appendix. Our experiments used CFR.

Maxmargin solving is safe. Furthermore, it guarantees that if every Player 1 best response reaches the subgame with positive probability through some infoset(s) that have positive margin, then exploitability is strictly lower than that of the blueprint strategy. While the theoretical guarantees are stronger, Maxmargin may lead to worse practical performance relative to Resolving when combined with the techniques discussed in Section 6, due to Maxmargin’s greater tendency to overfit to assumptions in the model.

Appendix B Description of Gadget Game

Solving the augmented subgame described in Maxmargin solving and Reach-Maxmargin solving will not, by itself, necessarily maximize the minimum margin. While LP solvers can easily handle this objective, the process is more difficult for iterative algorithms such as Counterfactual Regret Minimization (CFR) and the Excessive Gap Technique (EGT). For these iterative algorithms, the augmented subgame can be modified into a gadget game that, when solved, will provide a Nash equilibrium to the augmented subgame and will also maximize the minimum margin . This gadget game is unnecessary when using distributional alternative payoffs, which is introduced in section B.1.

The gadget game differs from the augmented subgame in two ways. First, all $P_{1}$ payoffs that are reached from the initial infoset of $I_{1}\in S_{r}$ are shifted by the alternative payoff of $I_{1}$, and there is longer an alternative payoff. Second, rather than the game starting with a chance node that determines $P_{1}$’s starting infoset, $P_{1}$ decides for herself which infoset to begin the game in. Specifically, the game begins with a $P_{1}$ node where each action in the node corresponds to an infoset $I_{1}$ in $S_{r}$. After $P_{1}$ chooses to enter an infoset $I_{1}$, chance chooses the precise node $h\in I_{1}$ in proportion to $\pi^{\sigma}_{-1}(h)$.

By shifting all payoffs in the game by the size of the alternative payoff, the gadget game forces $P_{1}$ to focus on improving the performance of each infoset over some baseline, which is the goal of Maxmargin and Reach-Maxmargin solving. Moreover, by allowing $P_{1}$ to choose the infoset in which to enter the game, the gadget game forces $P_{2}$ to focus on maximizing the minimum margin.

Figure 6 illustrates the gadget game used in Maxmargin and Reach-Maxmargin.

One problem with existing safe subgame-solving techniques is that they may “overfit” to the alternative payoffs, even when we use estimates. Consider for instance a subgame with two different $P_{1}$ infosets $I_{1}$ and $I^{\prime}_{1}$ at the top. Assume $P_{1}$’s value for $I_{1}$ is estimated to be $1$, and for $I^{\prime}_{1}$ is $10$. Now suppose during subgame solving, $P_{2}$ has a choice between two different strategies. The first sets $P_{1}$’s value in the subgame for $I_{1}$ to $0.99$ and for $I^{\prime}_{1}$ to $9.99$. The second slightly increases $P_{1}$’s value for the subgame for $I_{1}$ to $1.01$ but dramatically lowers the value for $I^{\prime}_{1}$ to . The safe subgame-solving methods described so far would choose the first strategy, because the second strategy leaves one of the margins negative. However, intuitively, the second strategy is likely the better option, because it is more robust to errors in the model. For example, perhaps we are not confident that $10$ is the exact value, but instead believe its true value is normally distributed with $10$ as the mean and a standard deviation of $1$. In this case, we would prefer the strategy that lowers the value for $I^{\prime}_{1}$ to .

To address this problem, we introduce a way to incorporate the modeling uncertainty into the game itself. Specifically, we introduce a new augmented subgame that makes subgame solving more robust to errors in the model. This augmented subgame changes the augmented subgame used in subgame Resolving (shown in Figure 3(b)) so that the alternative payoffs are random variables, and $P_{1}$ is informed at the start of the augmented subgame of the values drawn from the random variables (but $P_{2}$ is not). The augmented subgame is otherwise identical. A visualization of this change is shown in Figure 7. As the distributions of the random variables narrow, the augmented subgame converges to the Resolve augmented subgame (but still maximizes the minimum margin when all margins are positive). As the distributions widen, $P_{2}$ seeks to maximize the sum over all margins, regardless of which are positive or negative.

This modification makes the augmented subgame infinite in size because the random variables may be real-valued and $P_{1}$ could have a unique strategy for each outcome of the random variable. Fortunately, the special structure of the game allows us to arrive at a $P_{2}$ Nash equilibrium strategy for this infinite-sized augmented subgame by solving a much simpler gadget game.

The gadget game is identical to the augmented subgame used in Resolve subgame solving (shown in Figure 3(b)), except at each initial $P_{1}$ infoset $I_{r,1}\in S_{r}$, $P_{1}$ chooses action $a^{\prime}_{S}$ (that is, chooses to enter the subgame rather than take the alternative payoff) with probability P\big{(}X_{I_{1}}\leq v(I_{r,1},a^{\prime}_{S})\big{)}, where $v(I_{r,1},a^{\prime}_{S})$ is the expected value of action $a^{\prime}_{S}$. (When solving via CFR, it is the expected value on each iteration, as described in CFR-BR ). This leads to Theorem 3, which proves that solving this simplified gadget game produces a $P_{2}$ strategy that is a Nash equilibrium in the infinite-sized augmented subgame illustrated in Figure 7.

Let $S^{\prime}$ be a Resolve augmented subgame and $S^{\prime}_{r}$ its root. Let $S$ be a Distributional augmented subgame similar to $S^{\prime}$, except at each infoset $I_{r,1}\in S_{r}$, $P_{1}$ observes the outcome of a random variable $X_{I_{1}}$ and the alternative payoff is equal to that outcome. If CFR is used to solve $S^{\prime}$ except that the action leading to $S^{\prime}$ is taken from each $I_{r,1}\in S^{\prime}_{r}$ with probability P\big{(}X_{I_{1}}\leq v^{t}(I_{r,1},a^{\prime}_{S})\big{)}, where $v^{t}(I_{r,1},a^{\prime}_{S})$ is the value on iteration $t$ of action $a^{\prime}_{S}$, then the resulting $P_{2}$ strategy $\sigma^{S^{\prime}}_{2}$ in $S^{\prime}$ is a $P_{2}$ Nash equilibrium strategy in $S$.

Another option which also solves the game but has better empirical performance relies on the softmax (also known as Hedge) algorithm . This gadget game is more complicated, and is described in detail in Appendix C. We use the softmax gadget game in our experiments.

The correct distribution to use for the random variables ultimately depends on the actual unknown errors in the model. In our experiments for this technique, we set X_{I_{1}}\sim\mathcal{N}\big{(}\mu_{I_{1}},s^{2}_{I_{1}}\big{)}, where $\mu_{I}$ is the blueprint value (plus any gifts). $s_{I_{1}}$ is set as the difference between the blueprint value of $I_{1}$, and the true (that is, unabstracted) counterfactual best response value of $I_{1}$. Our experiments show that this heuristic works well, and future research could yield even better options.

Appendix C Hedge for Distributional Subgame Solving

In this paper we use CFR with Hedge in $S_{r}$, which allows us to leverage a useful property of the Hedge algorithm to update all the infosets resulting from outcomes of $X_{I_{1}}$ simultaneously.Another option is to apply CFR-BR only at the initial $P_{1}$ nodes when deciding between $a^{\prime}_{T}$ and $a^{\prime}_{S}$. When using Hedge, action $a^{\prime}_{S}$ in infoset $I_{r,1}$ in the augmented subgame is chosen on iteration $t$ with probability $\frac{e^{\eta_{t}\hat{v}(I_{r,1},a^{\prime}_{S})}}{e^{\eta_{t}\hat{v}(I_{r,1},a^{\prime}_{S})}+e^{\eta_{t}\hat{v}(I_{r,1},a^{\prime}_{T})}}$. Where $\hat{v}(I_{r,1},a^{\prime}_{T})$ is the observed expected value of action $a^{\prime}_{T}$, $\hat{v}(I_{r,1},a^{\prime}_{S})$ is the observed expected value of action $a^{\prime}_{S}$, and $\eta_{t}$ is a tuning parameter. Since, action $a^{\prime}_{S}$ leads to identical play by both players for all outcomes of $X$, $\hat{v}(I_{r,1},a^{\prime}_{S})$ is identical for all outcomes of $X$. Moreover, $\hat{v}(I_{r,1},a^{\prime}_{T})$ is simply the outcome of $X_{I_{1}}$. So the probability that $a^{\prime}_{S}$ is taken across all infosets on iteration $t$ is

where $f_{X_{I_{1}}}(x)$ is the pdf of $X_{I_{1}}$. In other words, if CFR is used to solve the augmented subgame, then the game being solved is identical to Figure 3(b) except that action $a^{\prime}_{S}$ is always chosen in infoset $I_{1}$ on iteration $t$ with probability given by (3). In our experiments, we set the Hedge tuning parameter $\eta$ as suggested in : $\eta_{t}=\frac{\sqrt{\ln(|A(I_{1})|)}}{3\sqrt{VAR(I_{1})_{t}}\sqrt{t}}$, where $VAR(I_{1})_{t}$ is the observed variance in the payoffs the infoset has received across all iterations up to $t$. In the subgame that follows $S_{r}$, we use CFR+ as the solving algorithm.

Appendix D Scaling of Gifts

To retain the theoretical guarantees of Reach subgame solving, one must ensure that the gifts assigned to reachable subgames do not (in aggregate) exceed the original gift. That is, if $g(I_{1})$ is a gift at infoset $I_{1}$, we must ensure that $CBV^{\sigma^{*}_{2}}(I_{1})\leq CBV^{\sigma_{2}}(I_{1})+g(I_{1})$. In this paper we accomplish this by increasing the margin of an infoset $I^{\prime}_{1}$, where $I_{1}\sqsubseteq I^{\prime}_{1}$, by at most $g(I_{1})$. However, empirical performance may improve if the increase to margins due to gifts is scaled up by some factor. In most games we experimented on, exploitability decreased the further the gifts were scaled. However, Figure 8 shows one case in which we observe the exploitability increasing when the gifts are scaled up too far. The graph shows exploitability when the gifts are scaled by various factors. At 0, the algorithm is identical to Maxmargin. at 1, the algorithm is the theoretically correct form of Reach-Maxmargin. Optimal performance in this game occurs when the gifts are scaled by a factor of about $1,000$. Scaling the gifts by $100,000$ leads to performance that is worse than Maxmargin subgame solving. This empirically demonstrates that while scaling up gifts may lead to better performance in some cases (because an entire gift is unlikely to be used in every subgame that receives one), it may also lead to far worse performance in some cases.

Appendix E Rules for Poker Variants

Our experiments are conducted on heads-up no-limit Texas hold’em (HUNL), as well as smaller-scale variants we call no-limit flop hold’em (NLFH) and no-limit turn hold’em (NLTH). We begin by describing the rules of HUNL.

In the form of HUNL discussed in this paper, each player starts a hand with $20,000. One player is designated$P_{1}$, while the other is$P_{2}$. This assignment alternates between hands. HUNL consists of four rounds of betting. On a round of betting, each player can choose to either fold, call, or raise. If a player folds, that player immediately surrenders the pot to the opponent and the game ends. If a player calls, that players places a number of chips in the pot equal to the opponent’s contribution. If a player raises, that player adds more chips to the pot than the opponent’s contribution. A round of betting ends after a player calls. Players can continue to go back and forth with raises in a round until one of them runs out of chips.

If either player chooses to raise first in a round, they must raise a minimum of $100. If a player raises after another player has raised, that raise must be greater than or equal to the last raise. The maximum amount for a bet or raise is the remainder of that player’s chip stack, which in our model is$20,000 at the beginning of a game.

At the start of HUNL, both players receive two private cards from a standard 52-card deck. $P_{1}$ must place a big blind of $100 in the pot, while$P_{2}$must place a small blind of$50 in the pot. There is then a round of betting (the preflop), starting with $P_{2}$. When the round ends, three community cards are dealt face up between the players. There is then another round of betting (the flop), starting with $P_{1}$ this time. After the round of betting completes, another community card is dealt face up, and another round of betting commences starting with $P_{1}$ (the turn). Finally, one more community card is dealt face up, and a final betting round occurs (the river), again starting with $P_{1}$. If neither player folds before the final betting round completes, the player with the best five-card poker hand, constructed from their two private cards and the five face-up community cards, wins the pot. In the case of a tie, the pot is split evenly.

NLTH is similar to no-limit Texas hold’em except there are only three rounds of betting (the preflop, flop, and turn) in which there are two options for bet sizes. There are also only four community cards. NLFH is similar except there are only two rounds of betting (the preflop and flop), and three community cards.

We experiment with two versions of NLFH, one small and one large, which include only a few of the available actions in each infoset. The small game requires $1.1$ GB to store the unabstracted strategy as double-precision floats. The large game requires $4$ GB. NLTH requires $35$ GB to store the unabstracted strategy.

Appendix F Proof of Theorem 1

Assume $M_{r}^{\sigma^{S}}(I_{1})\geq 0$ for every infoset $I_{1}$ and assume $\pi_{1}^{BR(\sigma^{\prime}_{2})}(I^{*}_{1})>0$ for some $I^{*}_{1}\in S_{\textit{top}}$ and let $\epsilon=M_{r}(I^{*}_{1})$. Define $\pi^{\sigma}_{-1}(I_{1})=\sum_{h\in I_{1}}\pi^{\sigma}_{-1}(h)$ and define $\pi^{\sigma}_{-1}(I_{1},I^{\prime}_{1})=\sum_{h\in I_{1},h^{\prime}\in I^{\prime}_{1}}\pi^{\sigma}_{-1}(h,h^{\prime})$.

We show that for every $P_{1}$ infoset $I_{1}\sqsubseteq I^{*}_{1}$ where $P(I_{1})=P_{1}$,

By the definition of $M^{\sigma^{S}}_{r}(I^{*}_{1})$ this holds for $I^{*}_{1}$ itself. Moreover, the condition holds for every other $I_{1}\in S_{\textit{top}}$, because by assumption every margin is nonnegative and $\pi_{-1}^{\sigma_{2}}(I_{1},I^{*}_{1})=0$ for any $I_{1}\in S_{\textit{top}}$ where $I_{1}\neq I^{*}_{1}$. The condition also clearly holds for any $I_{1}$ with no descendants in $S$ because then $\pi_{-1}^{\sigma_{2}}(I_{1},I^{*}_{1})=0$ and $\sigma^{\prime}_{2}(h)=\sigma^{-S}_{2}(h)$ in all $P_{2}$ nodes following $I_{1}$. This satisfies the base step. We now move on to the inductive step.

Let $Succ(I_{1},a)$ be the set of earliest-reachable $P_{1}$ infosets following $I_{1}$ such that $P(I^{\prime}_{1})=P_{1}$ for $I^{\prime}\in Succ(I_{1},a)$. Formally, $I^{\prime}_{1}\in Succ(I_{1},a)$ if $P(I^{\prime}_{1})=P_{1}$ and $I_{1}\cdot a\sqsubseteq I^{\prime}_{1}$ and for any other $I^{\prime\prime}_{1}\in Succ(I_{1},a)$, $I^{\prime\prime}_{1}\not\sqsubset I^{\prime}_{1}$. Then