On Regret-Optimal Learning in Decentralized Multi-player Multi-armed Bandits

I Introduction

Multi-armed bandit (MAB) models represent an exploration versus exploitation trade-off where the player must choose between exploring the environment to find better options, and exploiting based on her current knowledge to maximize her utility. These models are widely applicable in many application like display advertisements, sensor networks, route planning and spectrum sharing. The model can be understood through a simple game of choosing between two coins with unknown biases. The coins are tossed repeatedly and one of them is chosen at each instant. If at a given instance, the chosen coin turns up heads, we get a reward of $1, otherwise we get no reward. It is known that one of the two coins has a better bias, but the identity of the coin is not known. The question is, what is the optimal ‘learning’ policy that helps maximize the expected reward, i.e., to discover which coin has a better bias and at the same time maximize the cumulative reward as the game is played. Note that the player doesn’t know the value of the biases as well as she has no prior probability distribution on these values. This motivates the non-Bayesian setting. The formulation where the player has a prior distribution on the parameters is called Bayesian multi-armed bandits.

The idea of multi-armed bandit models dates back to Thompson and the first rigorous formulation is due to Robbins . The single player multi-armed bandit problem in a non-Bayesian setting was first formulated by Lai and Robbins . Any bandit policy that makes the best choice more than a constant fraction of the time is said to have sublinear regret. Regret measures the performance of any strategy formally against the best policy that could be employed if the distribution parameters were known. It was shown in that there is no learning policy that asymptotically has expected regret growing slower than $O(\log T)$. A learning scheme was also constructed that asymptotically achieved this lower bound.

This model was subsequently studied and generalized by many researchers. In , Anantharam et al. generalized it to the case of multiple plays, i.e., the player can pick multiple arms (or coins) when there are more than 2 arms. In , Agrawal proposed a sample mean based index policy that asymptotically achieved $O(\log T)$ regret. For the special case of bounded support for rewards, Auer et al. introduced a simple index-based policy, ${\tt UCB}_{1}$, that achieved logarithmic expected regret over finite time horizons. $\tt UCB_{1}$ has since become the benchmark to compare new algorithms against because of its power and simplicity.

Recently, policies based on Thompson Sampling (TS) have experienced a surge of interest due to their much better empirical performance . It is a probability matching policy which, unlike the $\tt UCB$-class of policies that use a deterministic confidence bound, draws samples from a distribution to determine which arm to play based on the probability of its being optimal. The logarithmic regret performance of the policy was not proved until very recently . introduced the Bayes-UCB algorithm which also uses use a Bayesian approach for analyzing the regret bound for stochastic bandit problems.

Deterministic sequencing algorithms which have separate exploration and exploitation phases have also appeared in the literature as an alternative to the joint exploration and exploitation approaches of UCB-like and probability matching algorithms. Noteworthy among these are the Phased Exploration and Greedy Exploitation policy for linear bandits that achieves $O(\sqrt{T})$ regret in general and $O(\log(T))$ regret for finitely many linearly parametrized arms. Other noteworthy algorithms include the logarithmic regret achieving deterministic sequencing of exploration and exploitation policy, with i.i.d. setting and with Markovian setting. Single-player bandit problems have also been looked at in the PAC framework, for instance, in . However, we restrict our attention to performance in the expected sense in this work.

In addition to single player bandits, there has been growing interest in multiplayer learning in multi-armed bandits, motivated by distributed sensor networks, wireless spectrum sharing and in particular cognitive radio networks. Suppose there are two wireless users trying to choose between two wireless channels. Each wireless channel is random, and looks different to each user. If channel statistics were known, we would try to determine a matching wherein the expected sum-rate of the two users is maximized. But the channel statistics are unknown, and they must be learnt by sampling the channels. Moreover, the two users have to do this independently and cannot share their observations as there is no dedicated communication channel between them. They, however, may communicate implicitly for coordination but this would come at the expense of reduced opportunities for rewards or benefits, and thus would add to regret. One can easily imagine a more general network setting with $M$ users and $N$ channels. This immediately gives rise to two questions. First, what is the lower bound for decentralized learning? That is, is there an inherent cost of decentralization in such network? And second, can we design a simple learning algorithm with provably optimal performance guarantees, in the context of such a decentralized network problem?

Policies for decentralized learning with sublinear regret have appeared in the literature for various models. When arms were restricted to have the same rewards for different users, Anandkumar et al. showed that logarithmic regret was achievable as the problem reduces to a ranking problem that can be solved in constant time in a decentralized manner. Similar works have also appeared for i.i.d. and Markovian arm reward settings. Relaxing this assumption makes the problem more complicated as it now becomes a bipartite matching problem and no decentralized algorithm performs quick enough. In our previous work , we proposed a policy, $\tt dUCB_{4}$ that achieved $O(\log^{2}T)$ regret through a recurrent negotiation mechanism between players. However, the answers to the two questions above remained unknown. In a similar work , authors address the problem of decentralized multi-armed bandits. While they address the same problem as ours, the emphasis is on the stability of this decentralized setting with minimum possible communication. Also, they don’t provide any optimality guarantees as compared to the optimal centralized learning problem. However, our paper assumes that players in the system remains the same. In users can arrive and leave at random times. Landgren et. al. uses a multi-armed bandit model for cooperative decision making problem in the context of running a consensus algorithm. Their setting is very different from the problem considered in this paper.

In this paper, we do not present an information theoretic lower bound on decentralized learning in a multiplayer multi-armed bandit problems. Such a result would be very interesting as it will also yield insight into the exact role of information sharing between players for a decentralized policy to work without an increase in expected regret. However, we managed to partially answer both questions above through two new decentralizable policies, $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$, where $\tt E^{3}$ stands for Exponentially-spaced Exploration and Exploitation policy, which we also call as $\tt E$-$\tt cubed$.

Both policies yield expected regret of the order $O(\log^{1+\delta}T)$ ($O(\log T)$ under some assumptions) in both single and multiplayer settings. The policies are based on exploration and exploitation in pre-determined phases such that over a long time horizon $T$, there are only logarithmically many slots in the exploration phases. It is well known that the optimal order of regret that can be achieved is $O(\log T)$ . These policies suggest an answer to the fundamental question of inherent cost to decentralize, that there is no cost to the order optimality, at least up to an $\log^{\delta}T$ factor. An asymptotic lower bound for the decentralized MAB problem (similar to that of the centralized MAB in ) is an important future research question.

The policies introduced in this paper, and the corresponding results hold even when the rewards are Markovian. However, we only present the i.i.d. case here and refer readers to our earlier paper for ideas on extensions to the Markovian setting. Extensive simulations were conducted to evaluate the empirical performances of these policies and compared to prior work in the literature, including the classical $\tt UCB_{1}$ and TS policies. The decentralized policies $\tt dE^{3}$ and $\tt dE^{3}$-$\tt TS$ are compared with the previously known $\tt dUCB_{4}$ policy.

The rest of the paper is organized as follows. Section II describes the model and problem formulations for single and multiplayer bandits. Section III describes relevant prior work in the area. The new policies $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$, and their multiplayer counterparts, $\tt dE^{3}$ and $\tt dE^{3}$-$\tt TS$ are described and studied in Section IV. Section V presents empirical performances of new and previous policies.

II Model and problem formulation

In this section, we describe problem formulations for single and multiplayer bandits. The single player formulation has been well-studied in literature, for instance, by Auer et al. and others. The multiplayer formulation is much newer, and has appeared in our previous work .

We consider an $N$-armed bandit problem. At each instant $t$, an arm $k$ is chosen, and a reward $X_{k}(t)$ is generated, from an independent and identically distributed (i.i.d.) random process with a fixed but unknown distribution. The processes are assumed to have bounded support, without loss of generality, in $$. Arm reward distributions have means$\mu_{k}$that are unknown. When choosing an arm, the player has access to the history of rewards and actions,$\mathcal{H}(t)$, with$\mathcal{H}(0):=\emptyset$. Denote the arm chosen at time$t$by$a(t)\in\mathcal{A}:=\{1,...,N\}$. A policy$\alpha$is a sequence of maps$\alpha(t):\mathcal{H}(t)\rightarrow\mathcal{A}$that specifies the arm chosen at time$t$. The player’s objective is to choose a policy that maximizes the expected reward over a finite time horizon$T$.

where arm $1$ is taken to have the greatest mean w.l.o.g.

In practical implementations of bandit algorithms in low-power settings such as sensor networks where the implementation of any learning/control policy should consume minimum amount of energy, it will be useful to include a computation cost as well. This is particularly the case when the algorithms must solve combinatorial optimization problems that are NP-hard. Such costs arise in decentralized settings in particular, where algorithms pay a communication cost for coordination between the decentralized players. For example, as we shall see later in our decentralized learning algorithm, the players may have to spend many time slots for coming up with a bipartite matching. We model it as a constant $C$ units of cost each time an index is computed by the policy. With this refinement, the regret of a policy $\alpha$ that computes its indices $m(T)$ times over a time horizon $T$ is,

where $n_{j}(T)$ is the number of times arm $j$ is played.

II-B Multiplayer model

We now describe the generalization of the single player, where we consider an $N$-armed bandit with $M$ players. We will refer to arms as channels interchangeably. There is no dedicated communication channel for coordination among the players. However, we do allow players to communicate with one another by playing arms in a certain way, e.g., arm 1 signals a bit ‘0’, arm 2 can signal a bit ‘1’. This of course will add to regret, and hence such communication comes at a cost. We assume that $N\geq M$.

At any instant $t$, each player choose one arm from the set of $N$ arms or takes no action (i.e., selects no arm). If more than one player picks the same arm, we regard it as a collision and this interference results in zero reward for those players. The rest of the model is similar to the single player case. Arm $k$ chosen by player $i$ generates an i.i.d. reward $S_{i,k}(t)$ from an unknown distribution, which has bounded support, w.l.o.g., in $$. Let$\mu_{i,k}$denote the unknown mean of$S_{i,k}(t)$.

Let $X_{i,k}(t)$ be the reward that player $i$ gets from playing arm $k$ at time $t$. Thus, if there is no collision, $X_{i,k}(t)=S_{i,k}(t)$. Denote the action of player $i$ at time $t$ by $a_{i}(t)\in\mathcal{A}:=\{1,\ldots,N\}$. Let $Y_{i}(t)$ be the communication message from player $i$ at time $t$ and $Y_{-i}(t)$ be the messages from all the other players except player $i$ at time $t$. Then, the history seen by player $i$ at time $t$ is $\mathcal{H}_{i}(t)=\{(a_{i}(1),X_{i,a_{i}(1)}(1),Y_{-i}(1)),\cdots,(a_{i}(t-1),X_{i,a_{i}(t-1)}(t-1),Y_{-i}(t-1))\}$ with $\mathcal{H}_{i}(0)=\emptyset$. A policy $\alpha_{i}=(\alpha_{i}(t))_{t=1}^{\infty}$ for player $i$ is a sequence of maps $\alpha_{i}(t):\mathcal{H}_{i}(t)\to\mathcal{A}$ that specifies the arm to be played at time $t$.

When expected rewards are not known, players must pick learning policies that minimize the expected regret, defined for policies $\alpha=(\alpha_{i},1\leq i\leq M)$ as,

As in the single player model, we consider a refinement of the regret to factor in computational or communication costs. Communication costs are justified because known distributed algorithms for bipartite matching require a certain amount of information exchange over multiple time slots. This cost will depend on the specific algorithm. Here, however, we will just consider an ‘abstract’ cost $C$.

Let $C$ units of cost be incurred each time this occurs, and let $m(t)$ be the number of times it happens in time $t$. Then, the expected regret for policy $\alpha$ to be minimized is,

where $\mathbf{k}^{**}$ is the optimal matching as defined in (3).

III Prior work

We now briefly describe the key features and results of existing single and multiplayer bandit policies.

We focus on three different MAB algorithms that capture different classes of policies.

In , Auer et al. proposed an index based policy, $\tt UCB_{1}$ which achieves logarithmic regret. It worked by playing the arm with the largest value of sample mean plus a confidence bound. The interval shrank deterministically as the arm got played more often and traded-off exploration and exploitation. It was shown in that the expected regret incurred by the policy over a horizon $T$ is bounded by,

Thompson Sampling (TS) is a probability-matching policy that has been around for quite some time in the literature although it was not well-studied in the context of bandit problems until quite recently . Arms are played randomly according to the probability of their being optimal. As an arm gets played more often, its sampling distribution become narrower. Unlike a fully Bayesian method such as Gittins Index , TS can be implemented efficiently in bandit problems. The regret of the policy was shown in to be bounded by,

where the constants have been omitted for brevity. A stronger upper bound for the case of Bernoulli rewards that matches the asymptotic rate lower bound in Lai and Robbins is given in . Numerically, TS has been found to empirically outperform $\tt UCB_{1}$ in most settings .

$\tt UCB_{4}$ is another confidence-bound based index policy that was proposed recently to overcome some of the shortcomings of the $\tt UCB_{1}$ policy, namely its reliance on index computation in each time step and the difficulty in extending the algorithm to a multiplayer setting. It works by cleverly choosing a sequence of times to compute the $\tt UCB_{1}$ index. The expected regret was shown in to be bounded by,

It can be shown that $\tt UCB_{1}$ and TS incur linear regret if computation cost is included in the model. The expected regret of the $\tt UCB_{4}$ algorithm over a time horizon $T$ with computation cost $C$ is bounded by ,

Thus, expected regret is $O(\log^{2}(T))$.

III-B Multiplayer policies

The major issues that are encountered in decentralizing bandit policies are coordination among players and finite precision of indices being communicated. The $\tt dUCB_{4}$ policy was the first such policy that did not assume identical channel rewards for different players. The policy is a natural decentralization of $\tt UCB_{4}$ that uses Bertsekas’ auction algorithm for distributed bipartite matching.

If $\Delta_{\min}$ is known, the expected regret of ${\tt dUCB_{4}}$ is,

IV New (near-)logarithmic bandit policies

In this section, we present our work in developing two closely related policies for single player bandit problems and their generalizations to multiplayer settings.

$\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ are phased policies detailed in Algorithms 1 and 2 respectively. Their key difference from the previous policies is that they have deterministic exploration and exploitation phases. In the following, an epoch is defined to comprise of one exploration phase and one exploitation phase.

Exploration phase: During an exploration phase, the player tries out different arms in a round-robin fashion and computes indices for each arm. At the end of the phase, the player chooses the arm with the maximum value of the index. The index computation differs for $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ policies.

Exploitation phase: In this phase, the player plays the arm that was chosen at the end of the previous exploration phase. No index computation happens during the exploitation phase and the player sticks to her decisions during this phase. The length of the exploitation phase doubles each successive epoch.

$\tt E^{3}$ and $\tt E^{3}$-$\tt TS$, while largely similar, differ in how they choose the arm to play during the exploitation phase. While $\tt E^{3}$ uses the simple sample mean value, $\tt E^{3}$-$\tt TS$ draws from a $\beta$-distribution in a manner similar to the TS policy.

The $\beta$-distribution is chosen in $\tt E^{3}$-$\tt TS$ due to convenient posterior form after Bernoulli observations. A $\beta(a,b)$-distribution prior results in a posterior of $\beta(a+1,b)$ or $\beta(a,b+1)$ depending on success or failure of the Bernoulli trial, respectively.

We now give the performance bounds for the policies with an index computation cost $C$ in the main result of this section. Both algorithms will be analyzed concurrently as their proof techniques are largely similar.

The following concentration inequality will be used in the analysis and is introduced here for the reader’s ease.

We now give the main result of this section.

(Regret bounds for $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ policies)

Let $\Delta_{\min}$ and $\Delta_{\max}$ denote the differences between the mean rewards of the optimal arm, and the second best and worst arms, respectively.

(i) If $\Delta_{\min}$ is known, set $\gamma=\lceil\frac{2}{\Delta_{\min}^{2}}\rceil$ and $\gamma_{\beta}=\lceil\frac{8}{\Delta_{\min}^{2}}\rceil$. Then, the expected regret of the $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ policies with computation cost $C$ is,

(ii) If $\Delta_{\min}$ is not known, choose $\gamma=\gamma_{t}$, where $\{\gamma_{t}\}$ is a positive sequence such that $\gamma_{t}\rightarrow\infty$ as $t\rightarrow\infty$. Then,

where $B(\delta)=2^{l(\delta)},l(\delta)=(\Delta^{2}_{\min}/4)^{-1/\delta}$.

(i) For the sake of clarity, we will assume that $\gamma_{t}$ changes at the beginning of every exploration phase. (ii) Part 1 of the above theorem assumes the knowledge $\Delta_{\min}$ in order to define $\gamma$. In fact we only need to know a lower bound on $\Delta_{\min}$. If $\Delta_{LB}\leq\Delta_{\min}$, we can fix $\gamma=\lceil\frac{2}{\Delta_{LB}^{2}}\rceil$. It is straightforward to show that, with a slight modification of the proof, the theorem still holds. Obviously, a tighter lower bound on $\Delta_{\min}$ results in a tighter bound on the regret.

Although the bounds of $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ are poorer than $\tt UCB_{1}$ and $\tt TS$, they lend themselves to easy decentralization and can be extended to multiplayer bandit problems with minimal effort. Performances of all single player algorithms are compared in Section V-A.

In this section, we present multiplayer generalizations of the $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ policies that were described in the previous section. They are detailed in Algorithms 3 and 4, respectively. They are also divided into exploration and exploitation phases.

Exploration phase: During exploration phases, players take turns to explore arms in a round-robin fashion. At the end of an exploration phase, the players update their index values (either $g_{i,j}$, or $\theta_{i,j}$). Then they participate in a distributed bipartite matching to determine the players to channels assignments. This requires some additional time slots and comes at a cost, and contributes to regret. This communication and the distributed bipartite matching process is compressed into line 5 in Algorithm 3 and line 8 in Algorithm 4 as a call to $\tt dBM$.

Distributed bipartite matching (dBM): Let $g(t)$ $(g_{i,j}(t),1\leq i\leq M,1\leq j\leq N)$ denote a vector of indices. In both algorithms, $\tt dBM_{\epsilon}$ $(g(t))$ refers to an $\epsilon$-optimal distributed bipartite matching algorithm, such as Bertsekas’ auction algorithm , that yields a matching $k^{*}(t)=(k_{1}^{*}(t),\ldots,k_{M}^{*}(t))\in\mathcal{P}(N)$ such that $\sum_{i=1}^{M}g_{i,k^{*}_{i}(t)}(t)\geq\sum_{i=1}^{M}g_{i,k_{i}}(t))-\epsilon,~{}\forall\mathbf{k}\in\mathcal{P}(N),\mathbf{k}\neq\mathbf{k}^{*}$. The details of $\tt dBM$ implementation is described in Section IV-D

Exploitation phase: In this phase, players stick to the allocation given to them at the end of the distributed bipartite matching process. No index computation is carried out in this phase. The length of the exploitation phase doubles in each successive epoch.

IV-C Regret analysis

We now give the main results of this section.

(i) Let $\epsilon>0$ be the precision of the bipartite matching algorithm and the precision of the index representation. If $\Delta_{\min}$ is known, choose $\epsilon$ such that $0<\epsilon<\Delta_{\min}/(M+1)$, set $\gamma=\left\lceil 2M^{2}/(\Delta_{\min}-(M+1)\epsilon)^{2}\right\rceil$ and $\gamma_{\beta}=\left\lceil 8M^{2}/(\Delta_{\min}-(M+1)\epsilon)^{2}\right\rceil$. Then, the expected regrets of the ${\tt dE^{3}}$ and $\tt dE^{3}$-$\tt TS$ policies are,

(ii) If $\Delta_{\min}$ is not known, choose $\gamma=\gamma_{t}$, where $\{\gamma_{t}\}$ is a positive sequence such that $\gamma_{t}\rightarrow\infty$ as $t\rightarrow\infty$. Also choose $\epsilon=\epsilon_{t}$, where $\{\epsilon_{t}\}$ is a positive sequence such that $\epsilon_{t}\rightarrow 0$ as $t\rightarrow\infty$. Then,

where $B(\delta)=b_{0}2^{l(\delta)},l(\delta)=(\Delta^{2}_{\min}/4)^{-1/\delta}$ and $b_{0}$ is a constant independent of $\delta$.

IV-D Distributed Bipartite Matching

Both $\tt dE^{3}$ algorithm and $\tt dE^{3}$-$\tt TS$ algorithm use the distributed bipartite matching algorithm as a subroutine. In Section IV-B we have given an abstract description of this distributed bipartite matching algorithm. We now present one such algorithm, namely, Bertsekas’ auction algorithm , and its distributed implementation. We note that the presented algorithm is not the only one that can be used. Both $\tt dE^{3}$ algorithm and $\tt dE^{3}$-$\tt TS$ algorithm will work with a distributed implementation of any bipartite matching algorithm, e.g. algorithms given in .

Consider a bipartite graph with $M$ players on one side, and $N$ arms on the other, and $M\leq N$. Each player $i$ has a value $\mu_{i,j}$ for each arm $j$. Each player knows only his own values. Let us denote by $k^{**}$, a matching that maximizes the matching surplus $\sum_{i,j}\mu_{i,j}x_{i,j}$, where the variable $x_{i,j}$ is 1 if $i$ is matched with $j$, and 0 otherwise. Note that $\sum_{i}x_{i,j}\leq 1,\forall j$, and $\sum_{j}x_{i,j}\leq 1,\forall i$. Our goal is to find an $\epsilon$-optimal matching. We call any matching $k^{*}$ to be $\epsilon$-optimal if $\sum_{i}\mu_{i,k^{**}(i)}-\sum_{i}\mu_{i,k^{*}(i)}\leq\epsilon$.

Here, $\text{second.max}_{j}$ is the second highest maximum over all $j$. The best arm for a player $i$ is arm $j_{i}^{*}=\arg\max_{j}(\mu_{i,j}-p_{j})$. The winner $i_{j}^{*}$ on an arm $j$ is the one with the highest bid.

The following lemma in establishes that Bertsekas’ auction algorithm will find the $\epsilon$-optimal matching in a finite number of steps.

Given $\epsilon>0$, Algorithm 5 with rewards $\mu_{i,j}$, for player $i$ playing the $j$th arm, converges to a matching $k^{*}$ such that $\sum_{i}\mu_{i,k^{**}(i)}-\sum_{i}\mu_{i,k^{*}(i)}\leq\epsilon$ where $k^{**}$ is an optimal matching. Furthermore, this convergence occurs in less than $(M^{2}\max_{i,j}\{\mu_{i,j}\})/\epsilon$ iterations.

Our only assumption here is going to be that each user can observe a channel, and determine if there was a successful transmission on it, a collision, or no transmission, in a given time slot. This consists of $J$ rounds. In each round, users transmit in a round robin fashion, where she can signal her channel preferences using $\lceil{\log M}\rceil$ bits and bid values (difference of top two indices) using $\lceil{\log 1/\epsilon_{1}}\rceil$ bits. The number of rounds $J$ is chosen so that the ${\tt dBM}$ algorithm (based on Algorithm 5) returns an $\epsilon_{2}$-optimal matching. More details on this implementation is given in .

V Simulations

We conducted extensive simulations comparing the performances of the proposed policies with prior work. The results are presented in the respective sections below.

For the single player setting, we considered a four-armed bandit problem with rewards for arms drawn independently from Bernoulli distributions with means $0.1,~{}0.5,~{}0.6,~{}0.9$. The scenario was simulated over a fixed time horizon $T=2,000,000$ timeslots and the performance of the proposed single-player policies was evaluated. The performance of each policy was averaged over 10 sample runs and the results presented here. Different true means and distributions were also considered and they gave similar rankings for the algorithms. In the interest of space, those scenarios are not presented.

In Figure 1, the single player policies proposed in this paper, $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$, are compared with the benchmark $\tt UCB_{1}$ policy. $\Delta_{\min}$ is assumed to be known $(0.1)$ and, consequently, $\gamma$ is fixed. The bound for $\tt E^{3}$-$\tt TS$ is also shown with the dashed line. It can be observed that, although, all three policies have logarithmic order of regret performance in time, the new $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ policies perform slightly worse than the $\tt UCB_{1}$ policy. This is attributable to the deterministic exploration phase length which must take into account the worst-case scenario. However, as we shall see in the next section, this gives us a significant performance advantage in the multiplayer setting.

Note that in Figure 1, computation cost is assumed to be zero. If computation cost were included, $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ would retain their logarithmic regret performance. However, the cumulative regret of $\tt UCB_{1}$ would grow linearly, just as with $\tt TS$ .

V-B Multiplayer bandit policies

We now present the empirical performance of the proposed $\tt dE^{3}$ and $\tt dE^{3}$-$\tt TS$ policies. We consider a three-player, three-armed bandit setting. Rewards for each arm are generated independently from a Bernoulli distribution with means $0.2,~{}0.25,~{}0.3$ for player 1, $0.4,~{}0.6,~{}0.5$ for player 2 and $0.7,~{}0.9,~{}0.8$ for player 3. A time horizon spanning 20 epochs was considered. $\epsilon=0.001$ was used as the tolerance for the bipartite matching algorithm, which was done using $\tt dBM_{\epsilon}$, a distributed implementation of Bertsekas’ auction algorithm. The performance of each policy was averaged over 10 sample runs. $\gamma$ was set equal to $100$ for $\tt dE^{3}$ and $400$ for $\tt dE^{3}$-$\tt TS$ (see analysis for the reason for differing $\gamma$’s). A fixed per unit cost each time the distributed bipartite matching algorithm $\tt dBM$ is run, is included in the setting to model communication cost in the decentralized setting.

The plot of the growth of cumulative regret with time of $\tt dE^{3}$, $\tt dE^{3}$-$\tt TS$ and $\tt dUCB_{4}$ is shown in Figure 2. We can see that the logarithmic regret performance of $\tt dE^{3}$ and $\tt dE^{3}$-$\tt TS$ clearly outperforms the $\log^{2}T$-regret performance of our earlier $\tt dUCB_{4}$ policy . The dashed line curve is the theoretical upper bound on the performance of $\tt dE^{3}$-$\tt TS$.

VI Conclusion

We designed two closely related single player and multiplayer bandit policies that achieve logarithmic or near-logarithmic regret performance depending on the assumptions of the model. Both policies have deterministic exploration and exploitation phases, which make them well-suited to decentralization for use in the multiplayer setting.

Performances of these policies were compared to prior work in the literature. They were shown to outperform previous policies for multiplayer bandits, but not for the single player model due to the deterministic phases of these new policies. While we have approached logarithmic regret performance under certain assumptions in the multiplayer model, the question of whether a policy under truly general conditions can achieve fully logarithmic regret remains open.

References

Appendix A Proof of Theorem 1

for both $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ policies.

Let $T$ be in the $l_{0}$-th exploitation epoch. By construction, $T\geq N\gamma l_{0}+2^{l_{0}}-2$. Thus, $\log T\geq l_{0}$ and,

where $\Delta_{j}=\mu_{1}-\mu_{j}$. Also, using the definition of computation cost,

The following two lemmas bound the event probabilities above for the $\tt E^{3}$ and $\tt E^{3}$-$\tt TS$ policies.

For $\tt E^{3}$, with $\gamma=\lceil\frac{2}{\Delta_{\min}^{2}}\rceil$,

The event $\{\overline{X}_{1}(l)<\overline{X}_{j}(l)\}$ implies at least one of the following events:

Using the Chernoff-Hoeffding bound and choosing $\gamma=\lceil\frac{2}{\Delta_{min}^{2}}\rceil$, we get,

For $\tt E^{3}$-$\tt TS$, with $\gamma_{\beta}=\lceil\frac{8}{\Delta_{\min}^{2}}\rceil$,

Without loss of generality, we will assume the underlying reward distributions of the arms to have a Bernoulli distribution to simplify the analysis. This eliminates the need for line 5 in the $\tt E^{3}$-$\tt TS$ policy illustrated in Algorithm 2. However, this assumption can be relaxed without any change to the results.

As in Lemma 2, the event $\{\theta_{1}(l)<\theta_{j}(l)\}$ implies at least one of the events:

Let $m_{j}(l)$ denote the number of plays of arm $j$ during the exploration phases after the $l$-th exploration epoch, and let $s_{j}(l)$ be the number of successes ($r=1$) in these plays. Then, $\theta_{j}(l)$ is sampled from a $\beta(s_{j}(l)+1,m_{j}(l)-s_{j}(l)+1)$ distribution.

Additionally, let $A(l)$ denote the event $\{\frac{s_{j}(l)}{m_{j}(l)}<\mu_{j}+\frac{\Delta_{j}}{4}\}$. Then,

where the last inequality comes from the Chernoff-Hoeffding inequality and by noting that $\frac{s_{j}(l)}{m_{j}(l)}$ is a random variable with mean $\mu_{j}$. Also, $m_{j}(l)=\gamma_{\beta}l$.

Here, $F^{B}_{n,p}(x)$ is the cdf of the $\tt binomial(n,p)$ distribution. The equality in the second-to-last line comes from the fact that $F^{\beta}_{a,b}(x)=1-F^{B}_{a+b-1,x}(a-1)$, where $F^{\beta}_{a,b}(x)$ is the cdf of the $\tt\beta(a,b)$ distribution . The inequality on the last line is a standard inequality for binomial distributions.

But, by the Chernoff-Hoeffding inequality, it can be seen that $F^{B}_{n,p}(np-n\delta)\leq\exp(-2n\delta^{2})$. Thus,

Setting $\gamma_{\beta}:=\lceil\frac{8}{\Delta_{\min}^{2}}\rceil$ in (A-A) and (30), we get,

Continuing with the proof of Theorem 1, thus,

Suppose $t_{l}$ be the time $t$ at which $l$th exploration phase begins. For the clarity of explanation, we assume that $\gamma$ changes only in the beginning of an exploration phase. So, in the $l$th exploration phase, each arms is played $\gamma_{t_{l}}$ times in a round robin manner.

As in the proof given in the previous subsection, let $T$ be in the $l_{0}$th exploitation epoch. By construction, $T\geq N\sum^{l_{0}}_{l=1}\gamma_{t_{l}}+2^{l_{0}}-2$. Thus, $\log T\geq l_{0}$ and,

The second inequality is from the fact that $\gamma_{t}$ is a monotone increasing sequence.

The computation cost is same as before, i.e.,

where $b_{1}=\Delta^{2}_{\min}/2$. Since $\gamma_{t}\rightarrow\infty$ monotonically (and $\gamma_{t_{k}}\geq 1$), there exists an $l^{\prime}$ such that $b_{1}\sum^{l}_{k=1}\gamma_{t_{k}}\geq l,\forall l>l^{\prime}$. Then,

where $B$ is a finite constant, independent of $T$.

When $\gamma_{t_{l}}=\log^{\delta}t_{l}$ for $\delta\in(0,1)$, it is easy to see that $\gamma_{t_{l}}\geq l^{\delta},\forall l$. Then, $\sum^{l}_{k=1}\gamma_{t_{k}}\geq\sum^{l}_{k=1}k^{\delta}\geq\int^{l+1}_{x=1}(x-1)^{\delta}dx\geq 0.5l^{(1+\delta)}$. From this, $l^{\prime}=(2/b_{1})^{1/\delta}$. Then, we can get $B=B(\delta)=2^{l^{\prime}}$.

Appendix B Proof of Theorem 2

We first show that if $\Delta_{\min}$ is known, we can choose an $\epsilon<\Delta_{\min}/(M+1)$, such that ${\tt dE^{3}}$ and $\tt dE^{3}$-$\tt TS$ algorithms achieve a logarithmic regret growth with $T$. If $\Delta_{\min}$ is not known, we can pick a positive monotone sequence $\{\epsilon_{t}\}$ such that $\epsilon_{t}\to 0$, as $t\to\infty$. In a decentralized bipartite matching algorithm, the precision $\epsilon$ will depend on the amount of information exchanged.

The proof will be illustrated here only for the $\tt dE^{3}$ policy since the differences between it and the analysis of the $\tt dE^{3}$-$\tt TS$ policy are similar to those found in Theorem 1.

Let us denote the optimal bipartite matching with $\mathbf{k}^{**}\in\mathcal{P}(N)$ such that $\mathbf{k}^{**}\in\arg\max_{\mathbf{k}\in\mathcal{P}(N)}\sum_{i=1}^{M}\mu_{i,\mathbf{k}_{i}}$. Denote $\mu^{**}:=\sum_{i=1}^{M}\mu_{i,\mathbf{k}_{i}^{**}}$, and define $\Delta_{\mathbf{k}}:=\mu^{**}-\sum_{i=1}^{M}\mu_{i,\mathbf{k}_{i}},~{}\mathbf{k}\in\mathcal{P}(N)$.

Let $\Delta_{\min}=\min_{\mathbf{k}\in\mathcal{P}(N),\mathbf{k}\neq\mathbf{k}^{**}}\Delta_{\mathbf{k}}$ and $\Delta_{\max}=\max_{\mathbf{k}\in\mathcal{P}(N)}\Delta_{\mathbf{k}}$. We assume $\Delta_{\min}>0$.

Let $T$ be in the $l_{0}$th exploitation epoch. It follows that, $T\geq MN\gamma l_{0}+2^{l_{0}}-2$ and, hence, $\log T\geq l_{0}$. Then,

A suboptimal matching occurs in the $l$-th exploitation epoch if the event $\{\sum^{M}_{i=1}\overline{X}_{i,k^{**}_{i}}(l)<(M+1)\epsilon+\sum^{M}_{i=1}\overline{X}_{i,k^{*}_{i}}(l)\}$ occurs. If each index has an error of at most $\epsilon$, the sum of M terms may introduce an error of at most $M\epsilon$. In addition, the distributed bipartite matching algorithm ${\tt dBM_{\epsilon}}$ itself yields only an $\epsilon$-optimal matching. This accounts for the term $(M+1)\epsilon$ above.

The event $\left\{\sum^{M}_{i=1}\overline{X}_{i,k^{**}_{i}}(l)<(M+1)\epsilon+\sum^{M}_{i=1}\overline{X}_{i,k^{*}_{i}}(l)\right\}$ implies at least one of the following events

for $1\leq i\leq M,1\leq j\leq N$. By the Chernoff-Hoeffding bound, and then using the fact that $\gamma=\lceil 2M^{2}/(\Delta_{\min}-(M+1)\epsilon)^{2}\rceil$,

where $\gamma_{\beta}=\lceil 8M^{2}/(\Delta_{\min}-(M+1)\epsilon)^{2}\rceil$.

The proof is similar to the proof of the analogous case of Theorem 1, and is omitted.