Collaborative Learning of Stochastic Bandits over a Social Network

I Introduction

We introduce and study a collaborative online learning paradigm, wherein a group of agents connected through a social network are engaged in learning a stochastic Multi-Armed Bandit (MAB) problem. In this setting, a set of agents are connected by a graph, representing an information-sharing network among them. At each time, each agent (a node in the social network graph) chooses an action (or arm) from a finite set of actions, and receives a stochastic reward corresponding to the chosen arm, from an unknown probability distribution. In addition, each agent shares the action index and the corresponding reward sample instantaneously with its neighbours in the graph. The agents are interested in maximising (minimising) their net cumulative reward (regret) over time. When there is only one learning agent, our setting is identical to the classical multi-armed bandit problem, which is a widely-studied framework for sequential learning .

Our framework is motivated by scenarios that involve multiple decision makers acting under uncertainty towards optimising a common goal. One such example is that of a large-scale distributed recommendation system, in which a network of backend servers handles user traffic in a concurrent fashion. Each user session is routed to one of the servers running a local recommendation algorithm. Due to the high volume of recommendation requests to be served, bandwidth and computational constraints may preclude a central processor from having access to the observations from all sessions, and issuing recommendations simultaneously to them in real time. In this situation, the servers must resort to using low-rate information from their neighbours to improve their learning, which makes this a collaborative networked bandit setting.

Another application scenario is that of cooperative transportation routing with mobile applications that provide social network overlays, like Waze . A user in this system is typically interested in taking the fastest or most efficient route through a city, with her app offering a choice of routes, and also recording observations from past choices. In addition, users can also add other trusted users as friends, whose observations then become available as additional information for future decision making. The social network among the users thus facilitates local information exchange, which could help users optimise their future decisions (choices of routes) faster.

In our setting, the agents use their social network to aid their learning task, by sharing their action and reward samples with their immediate neighbours in the graph. It seems reasonable that this additional statistical information can potentially help the agents to optimize their rewards faster than they would if they were completely isolated. Indeed, several interesting questions arise in this collaborative learning framework. For example, how does the structure of the social network affect the rate at which the agents can learn? Can good learning policies for the single agent setting be extended naturally to perform well in the collaborative setting? Can agents exploit their ‘place’ in the network to learn more efficiently? Can ‘more ‘privileged’ agents (e.g., nodes with high degree or influence) help other agents learn faster? This work investigates and answers some of these questions analytically and experimentally.

We consider the collaborative bandit learning scenario, and analyse the total regret incurred by the agents (regret of the network) over a long but finite horizon $n$. Our specific contributions in this paper are as follows.

We first introduce and analyse the expected regret of the UCB-Network policy, wherein all the agents employ an extension of the celebrated UCB1 policy. In this case, we derive an upper bound on the expected regret of a generic network. The upper bound involves a graph-dependent constant, which is obtained as the solution to a combinatorial optimisation problem. We then specialize the upper bound to common network topologies such as the fully connected and the star graphs, in order to highlight the impact of the social network structure on the derived upper bound.

Second, we derive a universal lower bound on the expected regret of a generic network, for a large class of ‘reasonable’ policies. This lower bound is based on fundamental statistical limits on the learning rate, and is independent of the network structure. To incorporate the network structure, we derive another lower bound on the expected regret of a generic network, as a function of a graph dependent parameter. This bound holds for the class of non-altruistic and individually consistent (NAIC) policies, which includes appropriate extensions of well-studied single agent learning policies, such as UCB1 and Thompson sampling to a network setting. We then observe that the gap between the derived lower bound for the NAIC class of policies, and the upper bound of the UCB-Network policy can be quite large, even for a simple star networkOur special interest in star graphs is motivated by the fact that social networks often posses a hub-and-spoke structure, where the star is a commonly occurring motif..

Third, we consider the class of star networks, and derive a refined lower bound on the expected regret of a large star network for NAIC policies. We observe that this refined lower bound matches (in an order sense) the upper bound of the UCB-Network. We thus conclude that widely-studied sequential learning policies (NAIC) which perform well in the single agent setting, may perform poorly in terms of the expected regret of the network when used in a network setting, especially when the network is highly hierarchical.

Next, motivated by the intuition built from our bounds, we seek policies which can exploit the social network structure in order to improve the learning rates. In particular, for an $m$-node star network, we propose a Follow Your Leader (FYL) policy, which exploits the centre node’s role as an ‘information hub’. We show that the proposed policy suffers a regret which is smaller by a factor of $m$ compared to that of any NAIC policy. In particular, the network-wide regret for the star-network under the FYL policy matches (in an order sense) the universal lower bound on regret. This serves to confirm that using the centre node’s privileged role is the right information structure to exploit in a star network.

Finally, we extend the above insights to a generic network. To this end, we make a connection between the smallest dominating set of the network, and the achievable regret under the FYL policy. In particular, we show that the expected regret of the network is upper bounded by the product of the domination number and the expected regret of a single isolated agent.

In sum, our results on the collaborative bandit learning show that policies that exploit the network structure often suffer substantially lesser expected regret, compared to single-agent policies extended to a network setting.

I-B Related Work

There is a substantial body of work that deals with the learning of various types of single agent MAB problems . However, there is relatively little work on the learning of stochastic MAB problems by multiple agents. Distributed learning of a MAB problem by multiple agents has been studied in the context of a cognitive radio frame work in . Unlike these models, a key novelty in our model is that it incorporates information sharing among the agents since they are connected by a network. In , the authors assume that each player, in each round, has access to the entire history corresponding to the actions and the rewards of all users in the network – this is a special case of our generic user network model. In , the authors deal with the learning of adversarial MAB problem by multiple agents connected through a network.

The primary focus in is centralized learning, wherein an external agent chooses the actions for the users in the network. The learning of the stochastic MAB problem by multiple users has also been addressed from a game-theoretic perspective in ; the randomised algorithm proposed therein uses the parameters of the MAB problem, which are unknown to the algorithm in practice. In contrast, we propose deterministic algorithms that do not require these parameters.

In a class of MAB problems considered in , a sole learning agent receives side observations in each round from other arms, in addition to samples from the chosen arm. Another related paper is – here, the model consists of a single major bandit (agent) and a set of minor bandits. While the major bandit observes its rewards, the minor bandits can only observe the actions of the major bandit. However, the bandits are allowed to exchange messages with their neighbours, to receive the reward information of the major bandit. Clearly, the models described above are rather different from the setting we consider in this work.

Organization. We describe the system model in Section II. Section III presents the regret analysis of the UCB-Network policy. Lower bounds on the expected regret of the network under certain classes of policies are presented in Section IV. Section V presents the regret analysis of the FYL policy. Numerical results are presented in Section VI, and Section VII concludes the paper.

II System Model

We first briefly outline the single agent stochastic MAB problem. Let $\mathcal{K}=\{1,2,\dots,K\}$ be the set of arms available to the agent. Each arm is associated with a distribution, independent of others, say $\mathcal{P}_{1},\mathcal{P}_{2},\dots,\mathcal{P}_{K}$, and let $\mu_{1},\mu_{2},\dots,\mu_{K}$ be the corresponding means, unknown to the agent. Let $n$ be the time horizon or the total number of rounds. In each round $t$, the agent chooses an arm, for which he receives a reward, an i.i.d. sample drawn from the chosen arm’s distribution. The agent can use the knowledge of the chosen arms and the corresponding rewards upto round $(t-1)$ to select an arm in round $t.$ The goal of the agent is to maximize the cumulative expected reward up to round $n$.

The policy $\Phi^{v}$ followed by a user prescribes actions at each time $t,$ $\Phi^{v}(t):H^{v}(t)\rightarrow\mathcal{K},$ where $H^{v}(t)$ is the information available with the user till round $t.$ A policy of the network $G,$ denoted by $\Phi,$ comprises of the policies pertaining to all users in $G.$ The performance of a policy is quantified by a real-valued random variable, called regret, defined as follows. The regret incurred by user $v$ for using the policy $\Phi^{v}$ upto round $n$ is defined as,

where $a^{v}(t)$ is the action chosen by the policy $\Phi^{v}$ at time $t$, and $\mu^{*}=\max\limits_{1\leq i\leq K}\mu_{i}$. We refer to the arm with the highest expected reward as the optimal arm. The regret of the entire network $G$ under the policy $\Phi$ is denoted by $R^{G}_{\Phi}(n)$, and is defined as the sum of the regrets of all users in $G$. The expected regret of the network is given by:

where $\Delta_{i}=\mu^{*}-\mu_{i}$, and $T_{i}^{v}(n)$ is the number of times arm $i$ has been chosen by $\Phi^{v}$ upto round $n$. We omit $\Phi$ from the regret notation, whenever the policy can be understood from the context. Our goal is to devise learning policies in order to minimise the expected regret of the network.

Let $\mathcal{N}(v)$ denote the set consisting of the node $v$ and its one-hop neighbours. Let $m^{v}_{i}(t)$ be the number of times arm $i$ has been chosen by node $v$ and its one-hop neighbours till round $t$, and $\hat{\mu}_{m_{i}^{v}(t)}$ be the average of the corresponding reward samples. These are given as:

III The UCB-Network policy

Motivated by the well-known single agent policy UCB1 , we propose a distributed policy called the UCB-user. This is a deterministic policy, since, for a given action and reward history, the action chosen is deterministic. When each user in the network follows the UCB-user policy, we term the network policy as UCB-Network which is outlined in Algorithm 1.

The following theorem presents an upper bound on the expected regret of a generic network, under the UCB-Network policy.

Assume that the network $G$ follows the UCB-Network policy to learn a stochastic MAB problem with $K$ arms. Further, assume that the rewards lie in $$. Then,

The expected total regret of $G$ is upper bounded as:

where $\Delta_{i}=\mu^{*}-\mu_{i}$, $\beta\in(0.25,1)$, $b~{}=~{}m\left(\frac{2}{4\beta-1}+\frac{2}{(4\beta-1)^{2}\ln(1/\beta)}\right)\left(\sum\limits_{j=1}^{K}\Delta_{j}\right),$ and $C_{G}$ is a network dependent parameter, defined as follows.

Let $\gamma_{k}=\min\{t\in\{1,\dots,n\}:|\{v\in V:m^{v}_{i}(t)\geq l_{i}=\frac{8\ln n}{\Delta_{i}^{2}}\}|\geq k\}$ denote the smallest time index when at least $k$ nodes have access to at least $l_{i}$ samples of arm $i$. Let $\eta_{k}$ be the index of the ‘latest’ node to acquire $l_{i}$ samples of arm $i$ at $\gamma_{k},$ such that $\eta_{k}\neq\eta_{k^{\prime}}$ for $1\leq k,k^{\prime}\leq m$. Define $z_{k}=T_{i}(\gamma_{k}):=\left(T^{1}_{i}(\gamma_{k}),\dots,T^{m}_{i}(\gamma_{k})\right)$, which contains the arm $i$ counts of all nodes at time $\gamma_{k}$. Then, $C_{G}l_{i}$ is the solution of the following optimisation problem:

Interpretation of (2): Under the UCB-Network policy, suppose a node has acquired at least $l_{i}$ samples of a sub-optimal arm $i$. As shown in the Lemma 2 in the Appendix A that such a node will not play the sub-optimal arm $i$ subsequently with high probability. Next, note that, $z_{k}$ is a vector of arm $i$ counts (self plays) of all nodes at time $\gamma_{k}$. The objective function in (2) represents the sum of arm $i$ counts of all nodes at the smallest time index, when all nodes have access to at least $l_{i}$ samples of arm $i$. The solution to (2) represents the maximum number of samples of arm $i$ required by the entire network such that $(a)$ Each node has access to at least $l_{i}$ samples of arm $i$ (the last constraint in (2)), and $(b)$ Each node stops choosing arm $i$ after it has access to $l_{i}$ samples of it (the penultimate constraint in (2)).

For example, the solution to (2) for an $m$-node star network (shown in Fig. 1) is $(m-1)l_{i}$. This corresponds to the scenario where the center node never chooses the sub-optimal arm $i$, and each leaf node chooses it $l_{i}$ times.

Proof sketch: First, we show that any node $v$ plays any sub-optimal arm $i$ in a given round $t$ with small probability after it has $l_{i}$ samples of it, in Lemma 2. Using Lemma 2, we then upper bound the expected regret of the network after each node has $l_{i}$ samples of the sub-optimal arm $i$. Next, we upper bound the maximum number of samples of the sub-optimal arm $i$ required by the entire network such that each node has access to $l_{i}$ samples of it, in Lemma 3. Finally, we obtain the desired upper bound by combining Lemma 2 and Lemma 3. A detailed proof, along with Lemma 2 and 3 is given in the Appendix A.

Solving (2) for an arbitrary network is analytically complex. Hence, we solve the problem for a few specific networks that range from high connectivity to low connectivity; namely, the $m$-node Fully Connected (FC), circular, star and Fully Disconnected (FD) networks. For $m=5$, these networks are shown in Fig. 1. It is easy to verify that the solution to (2) for these four networks are $l_{i}$, $(m-1)l_{i}$, $\lfloor\frac{m}{2}\rfloor l_{i}$ and $ml_{i},$ respectively. We can then evaluate the upper bounds in Theorem 1. Corollary 1 For an $m$-node FC network:

Corollary 2 For an $m$-node circular network:

Corollary 3 For an $m$-node star network:

A key insight can be obtained from the above corollaries is that, the expected regret of a network decreases by a factor of $m$, $2$ and $m/(m-1)$ in the cases of $m$-node FC, circular and star networks respectively, compared to FD network.

IV Lower bounds on the expected regret

In this section, we derive lower bounds on the expected regret of the network under various classes of policies. Our first lower bound is a universal bound which is independent of the user network, and holds for large class of ‘reasonable’ learning policies. Second, we derive a network-dependent lower bound for a class of Non-Altruistic and Individually Consistent (NAIC) policies – a class that includes network extensions of well-studied policies like UCB1 and Thompson sampling. Finally, we derive a refined lower bound for large star networks under NAIC policies.

Throughout this section, we assume that the distribution of each arm is parametrised by a single parameter. We use $\boldsymbol{\theta}=\left(\theta_{1},\dots,\theta_{K}\right)\in\Theta^{K}=\boldsymbol{\Theta}$ to denote the parameters of arms $1$ to $K$ respectively. Suppose $f(x;\theta_{j})$ be the reward distribution for arm $j$ with parameter $\theta_{j}$. Let $\mu(\theta_{j})$ be the mean of arm $j$, and $\theta^{*}=\underset{1\leq j\leq K}{\arg\max}~{}\mu(\theta_{j})$. Define the parameter sets for an arm $j$ as

Note that $\boldsymbol{\Theta_{j}}$ contains all parameter vectors in which the arm $j$ is a sub-optimal arm, and $\boldsymbol{\Theta_{j}^{*}}$ contains all parameter vectors in which the arm $j$ is the optimal arm. Let $kl(\beta||\lambda)$ be the KL divergence of the distribution parametrised by $\lambda$, from the distribution parametrised by $\beta$.

[A1] We assume that the set $\Theta$ and $kl(\beta||\lambda)$ satisfy the following :

$f(.;.)$ is such that $0<kl(\beta||\lambda)<\infty$ whenever $\mu(\lambda)>\mu(\beta).$

$\forall\epsilon>0$ and $\forall\beta,\lambda$ such that $\mu(\lambda)>\mu(\beta),\exists\delta=\delta(\epsilon,\beta,\lambda)>0$ for which $|kl(\beta||\lambda)-kl(\beta||\lambda^{\prime})|<\epsilon$ whenever $\mu(\lambda)\leq\mu(\lambda^{\prime})\leq\mu(\lambda)+\delta.$

$\Theta$ is such that $\forall\lambda\in\Theta$ and $\forall\delta>0,\exists\lambda^{\prime}\in\Theta$ such that $\mu(\lambda)<\mu(\lambda^{\prime})<\mu(\lambda)+\delta.$

Note that the above universal lower bound is based on fundamental statistical limitations, and is independent of the network $G$. Next, we define the class of NAIC policies, and derive a network-dependent lower bound for this class. In the rest of this section, we assume that each arm is associated with a discrete reward distribution, which assigns a non-zero probability to each possible value.

Let $\omega$ be a sample path, which consists of all pairs of actions and the corresponding rewards of all nodes from rounds $1$ through $n$:

Definition 1 [Individually consistent policy] A policy followed by a user $v$ is said to be individually consistent if, for any sub-optimal arm $i$, and for any policy of a user $u\in\mathcal{N}(v)\setminus\{v\}$

Definition 2 [Non-altruistic policy] A policy followed by a user $v$ is said to be non-altruistic if there exist $a_{1},a_{2}$, not depending on time horizon $n$, such that the following holds. For any $n$ and any sub-optimal arm $i$, the expected number of times that the policy plays arm $i$ after having obtained $a_{1}\ln n$ samples of that arm is no more than $a_{2}$, irrespective of the policies followed by the other users in the network.

It can be shown that UCB-user and Thompson sampling are NAIC policies. In particular, we show that the UCB-user policy is an NAIC policy in Lemma 4 in Appendix A.

Note that the above policy, follow node $u$, is in fact a non-trivial and rather well-performing policy that we will revisit in Section V. We now derive a network-dependent lower bound for the class of NAIC policies

Let $G=(V,E)$ be a network with $m$ nodes, and suppose [A1] holds. If each node in $V$ follows an NAIC class policy to learn a $K$-arm stochastic MAB problem with a parameter vector of arms as $\boldsymbol{\theta}=(\theta_{1},\dots,\theta_{K})\in\boldsymbol{\Theta_{j}}$, and $\delta\in(0,1)$ then, the following lower bounds hold:

where $L_{G}$ can be obtained from the solution to the following optimisation problem:

The notation used in (10) is the same as the notation in Theorem 1, except that $l_{i}$ is replaced with $q_{j}$. Further, $L_{G}$ is obtained by dividing the solution to (10) by $q_{j}$. Similar to (2), solving (10) analytically for an arbitrary network is difficult. Hence, we focus on solving (10) for the networks shown in Fig. 1, and provide the corresponding lower bounds below. Let $\Delta_{i}=\mu(\theta^{*})-\mu(\theta_{i})$. Corollary 5 For an $m$-node FC network:

Corollary 6 For an $m$-node circular network:

Corollary 7 For an $m$-node star network:

From corollaries 1-8, we infer that the upper bound of the UCB-Network policy and the lower bound given by (9) are of the same order, for FC $(\ln n)$, circular $(m\ln n)$ and FD $(m\ln n)$ networks. However, for star networks, there is a large gap between the UCB-Network upper bound and the lower bound for NAIC policies in (13). Since the UCB-Network is an NAIC class policy, we proceed to ascertain if either of these bounds is too loose for star networks. Our special interest in star networks is due to the prevalence of hubs in many social networks, and as we shall see in the next section, this hierarchical structure can be exploited to enhance the learning rate.

Next, we consider a specific instance of a large star network, for which we derive a refined lower bound for the class of NAIC policies. This refined lower bound is of the same order as the regret upper bound for the UCB-Network policy, implying that the upper bound in Theorem 1 is tight in an order sense, and cannot be improved in general.

Let $G_{n}=(V_{n},E_{n})$ be a sequence of $m_{n}$-node star networks learning a $2$-arm stochastic MAB problem with mean rewards $\mu_{a},\mu_{b}$ such that $\mu_{a}>\mu_{b}$. Suppose $m_{n}\geq 2\cdot\frac{\ln n}{kl(\mu_{b}||\mu_{a})}$, and that each node follows an NAIC policy. Then,

We now briefly explain the intuition behind Theorem 4. In a large star network, the center node learns the sub-optimal arm very quickly (in a few rounds), since it has access to a large number of samples in each round. Under an NAIC policy, once a node has enough samples to learn that an arm is sub-optimal, by definition, it stops choosing that arm with high probability. Hence, the center node stops choosing the sub-optimal arm with high probability, which in turn ensures that the leaf nodes learn the sub-optimal arm themselves, by choosing the sub-optimal arm $O(\ln n)$ times. This leads to a regret of $O((m-1)\ln n)$. Our simulation results, in Table I, also illustrates this behaviour, for the UCB-Network policy (which is NAIC) on large star networks.

Theorem 4 asserts that, for a fixed, large time horizon $n$, we can construct a large star network with $m$ nodes, whose expected regret is atleast $O((m-1)\ln n)$. This lower bound matches with the upper bound for UCB-Network in Theorem 1. Thus, we conclude that the class of NAIC policies could suffer a large regret, matching the upper bound in an order sense. However, for the same star network and time horizon, the universal lower bound in (7) turns out to be $O(\ln n)$. This gap suggests the possibility that there might exist good learning policies (which are not NAIC) for a star network, with regret matching the universal lower bound. In the next section, we propose one such policy, which does not belong to the NAIC class.

V The Follow Your Leader (FYL) Policy

In this section, we first outline a policy called Follow Your Leader (FYL) for a generic $m$-node network. The policy is based on exploiting high-degree hubs in the graph; for this purpose, we define the dominating set and the dominating set partition.

Definition 3 [Dominating set of a graph] A dominating set $D$ of a graph $G=(V,E)$ is a subset of $V$ such that every node in $V\setminus D$ is adjacent to atleast one of the nodes in $D$. The cardinality of the smallest dominating set of $G$ is called as the domination number.

Definition 4 [Dominating set partition of a graph] Let $D$ be a dominating set of $G$. A dominating set partition based on $D$ is obtained by partitioning $V$ into $|D|$ components such that each component contains a node in $D$ and a subset of its one hop neighbors.

Note that given a dominating set for a graph, it is easy to obtain a corresponding dominating set partition. The FYL policy for an $m$-node generic network is outlined in Algorithm 2. Under the FYL policy, all nodes in the dominating set are called leaders and all other nodes as followers; the follower nodes follow their leaders while choosing an action in a round. As we argued in Section IV, the policy deployed by a follower node in FYL is not individually consistent. The following theorem presents an upper bound on the expected regret of an $m$-node star network which employs the FYL policy.

Suppose the star network $G$ with a dominating set as the center node, follows the FYL policy to learn a stochastic MAB problem with $K$ arms. Assume that the rewards lie in $$. Then,

where d=\Big{[}2m-1+\frac{2m}{4\beta-1}\left(1+\frac{1}{(4\beta-1)\ln(1/\beta)}\right)\Big{]}\sum\limits_{j=1}^{K}\Delta_{j}, $\Delta_{i}=\mu^{*}-\mu_{i}$ and $\beta\in(0.25,1)$.

A key insight obtained from Theorem 5 is that an $m$-node star network with the FYL policy incurs an expected regret that is lower by a factor $(m-1)$, as compared to any NAIC policy. More importantly, we observe that the regret upper bound under the FYL policy meets the universal lower bound in (7). Hence, we conclude that the FYL policy is order optimal for star networks.

Finally, we present a result that asserts an upper bound on the expected regret of a generic network under the FYL policy.

Let $D$ be a dominating set of an $m-$node network $G=(V,E).$ Suppose $G$ with the dominating set $D$ employs the FYL policy to learn a stochastic MAB problem with $K$ arms, and the rewards lie in $$, then

From the above theorem we infer that, the expected regret of a network scales linearly with the cardinality of a given dominating set. Hence, in order to obtain a tighter upper bound, we need to supply a smallest dominating set $D^{*}$ to the FYL policy. Suppose, if we provide $D^{*}$ as the input to the FYL policy, then we obtain an improvement of factor $m/|D^{*}|$ in the expected regret of an $m$-node network compared to the fully disconnected network.

It is known that, computing a smallest dominating set of a given graph is an NP-hard problem . However, fast distributed approximation algorithms for the same are well-known in the literature. For example, Algorithm $35$ in finds a smallest dominating set with an approximation factor $\log(\text{MaxDegree}(G)).$ Also, upper bounds on the domination number for specific networks such as Erdos-Renyi, power-law preferential attachment and random geometric graphs are available in .

VI Numerical Results

We now present some simulations that serve to corroborate our analysis. The simulations have been carried out using MATLAB, and are averaged over 100 sample paths. We fix the time horizon $n$ to be $10^{5}$.

We consider the following two scenarios: $(i)$ 10 node FC, circular, star and FD networks, 2 arms, Bernoulli rewards with means $0.7,0.5$, and $(ii)$ 20 node FC, circular, star and FD networks, 10 arms, Bernoulli rewards with means $1,0.9,0.8,\dots,0.1$. We run the UCB-Network policy for these scenarios, and calculate the expected regret of the network and percentage of time the optimal arm is played by the network. The results are shown in Fig. 4 and 4. It can be observed from Fig. 4 and 4 that the expected regret of the network decreases and the percentage of time the optimal arm is chosen by the network increases, as connectivity of the network increases. This is because, an increase in the connectivity of the network increases the number of observations available to a user, in a given round.

VI-B Performance of UCB-Network on star networks

We consider 5, 10, 25, 50, 100, 200 and 350 node star networks, each learning a 2-armed stochastic bandit problem with Bernoulli rewards of means 0.7 and 0.5. We run the UCB-Network policy on the aforementioned networks, and summarise the results in Table I. Observe that, the expected number of times the center node chooses arm 2 (sub-optimal arm) decreases as the network size increases. This forces each leaf node to choose arm 2 on its own in order to learn. Therefore, as the star network size increases, the expected regret of the network can be approximated as the product of the network size and the expected regret of an isolated node.

VI-C Comparison of UCB-Network and FYL policies

We consider 25, 100 and 350 node star networks learning a 2-arm stochastic bandit problem with Bernoulli rewards of means 0.7 and 0.5. We run both UCB-Network and FYL policies on the above-mentioned networks. It can be observed from Fig. 4 that the star networks incur much smaller expected regret under the FYL policy, as compared to UCB-Network, and learn the optimal arm much faster.

VII Concluding Remarks

We studied the collaborative learning of a stochastic MAB problem by a group of users connected through a social network. We analysed the regret performance of widely-studied single-agent learning policies, extended to a network setting. Specifically, we showed that the class of NAIC policies (such as UCB-Network) could suffer a large expected regret in the network setting. We then proposed and analysed the FYL policy, and demonstrated that exploiting the structure of the network leads to a substantially lower expected regret. In particular, the FYL policy’s upper bound on the expected regret matches the universal lower bound, for star networks, proving that the FYL policy is order optimal. This also suggests that using the center node as an information hub is the right information structure to exploit.

In terms of future research directions, we plan to study this model for other flavours of MAB problems such as linear stochastic and contextual bandits . Even in the basic stochastic bandit model considered here, several fundamental questions remain unanswered. For a given network structure, what is the least regret achievable by any local information-constrained learning strategy? Is it possible in a general network to outperform ‘good single-agent’ policies (i.e., those that work well individually, like UCB) run independently throughout the network? If so, what kind of information sharing/exchange might an optimal strategy perform? It is conceivable that there could be sophisticated distributed bandit strategies that could signal within the network using their action/reward sequences, which in turns begs for an approach relying on information-theoretic tools.

References

Appendix A

We require the following Lemma 1, 2, 3 and inequality to prove Theorem 1. Hoeffding’s Maximal Inequality : Let $X_{1},X_{2},\dots$ be centered i.i.d random variables lying in $$. Then, for any$x>0$and$t\geq 1$,

In order to introduce Lemma 1, we need the following.

We prove that the probability of a sample path of the network in both probability spaces are equal, in the following lemma. Hence, this allows us to equivalently work in the new probability space, as and when appropriate.

Consider an $m$-node undirected user graph. Let $A(t)$ and $Z(t)$ be the random variables which indicate the actions chosen by all nodes and the corresponding rewards, in round $t$. Let $E(k)=\left(A(k),Z(k),\dots,A(1),Z(1)\right)$. Then, $\forall t\geq 1$,

where $\bar{a}_{1:t}=\left(\bar{a}_{1},\dots,\bar{a}_{t}\right),\bar{z}_{1:t}=\left(\bar{z}_{1},\dots,\bar{z}_{t}\right)$ with $\bar{a}_{k}\in\mathcal{K}^{m}$ and $\bar{z}_{k}\in^{m}$ for any $k\geq 1$.

We establish the result using induction on $t$. The result trivially holds for $t=1$, since a policy does not possess any information in the very first round itself. Assume that it is true for $t=k$. Then,

Now, we prove that the result holds for $t=k+1$.

By substituting (Proof:) in (16), we obtain

Let $c_{t,S}=\sqrt{\frac{2\ln t}{S}},\beta\in(0,1)$. For each $v\in V$ and sub-optimal arm $i$, define $\tau^{v}_{i}$ as follows:

Observe that, the event $A^{v}_{i}(t)$ occurs only if atleast one of the following events occur.

Note that, the event given by (21) does not occur when the event $\{m^{v}_{i}(t)\geq l_{i}\}$ occurs. Hence,

For each node $v\in V$ and each arm $i$, the initialization phase of the UCB-user policy implies that $|\mathcal{N}(v)|\leq m^{v}_{i}(t)\leq|\mathcal{N}(v)|t$. Therefore,

Here, (23) is due to the peeling argument on geometric grid over $[|\mathcal{N}(v)|,|\mathcal{N}(v)|t]$. This implies that, for $\beta\in(0,1)$, $a\geq 1$, if $s\in\{a,\dots,at\}$ then there exists $j\in\{0,\dots,\frac{\ln t}{\ln\left(1/\beta\right)}\}$ such that $a\beta^{j+1}t<s\leq a\beta^{j}t$. Now, we proceed to bound the probability of the event given by (24) using Hoeffding’s maximal inequality and Lemma 1. Hence,

Substituting (25) and (26) in (22) gives the desired result. ∎

Let $\tau^{v}_{i}$ $\forall v\in V,$ and $l_{i}$, $\forall 1\leq i\leq K$ be as defined in the Lemma 2. Assume that a node $v$ stops playing the sub-optimal arm $i$ at time $\tau^{v}_{i}$. Then, for an arm $i$, $\sum\limits_{v\in V}T^{v}_{i}(\tau^{v}_{i})\leq C_{G}l_{i}$, where $C_{G}l_{i}$ is the solution to the optimisation problem in (2).

We first evaluate the value of the random variable $\sum\limits_{v=1}^{m}T^{v}_{i}(\tau^{v}_{i})$ for all realizations. Then, we determine the maximum value of the random variable over all realizations. The following algorithm gives the value of the above mentioned random variable for a realization. Consider an $m$ length column vector of zeros, say $y$. Algorithm: Step 1: Select an integer $I$ from $B=\{1,2,\dots,m\}$. Step 2: Increase $y(I)$ by 1, i.e., $y(I)=y(I)+1$. Step 3: Find the indices (say $C$) corresponding to elements in $Ay$ which are atleast $l_{i}$. Here, $A$ is the adjacency matrix of the graph $G$. Step 4: Update $B=B\setminus C$ and $A$ by removing rows corresponding to $C$ in $A$ Step 5: Go to step 1, if $B$ is non-empty else stop by returning $y$. Here, step 4 ensures that nodes having $l_{i}$ samples of arm $i$ stops playing arm $i$ further. Observe that $\|y\|_{1}$, where $y$ is the vector returned by the above algorithm, yields the value of the random variable $\sum\limits_{v=1}^{m}T^{v}_{i}(\tau^{v}_{i})$ for a realization. Therefore, it suffices to maximize $\|y\|_{1}$ over all realizations. The optimisation problem in $\eqref{OptimizationProblem11}$ captures the above. The final constraint in $\eqref{OptimizationProblem11}$ ensures that the node $\eta_{k}$ has $l_{i}$ samples of sub-optimal arm $i$ at time instance $\gamma_{k}$. Recall that, $\gamma_{k}$ is a random variable which tracks the least time at which atleast $k$ nodes have more than $l_{i}$ samples of arm $i$. The penultimate constraint ensures that sub-optimal arm $i$ count of node $\eta_{k}$ does not increase(or stop playing arm $i$) after time instance $\gamma_{k}$. Hence, a feasible point in the above optimisation problem is a sequence $\{z_{k}\}_{k=1}^{m}$ which satisfies the aforementioned two constraints. Then, $\|z_{m}\|_{1}$ corresponds to the value of the random variable $\sum\limits_{v=1}^{m}T^{v}_{i}(\tau^{v}_{i})$ for a realization. ∎

By using the above lemmas, we now prove Theorem 1.

Now, we upper bound $(b)$ in (28). Let $1\leq v\leq m$. Since, $m^{v}_{i}(t)\geq l_{i}$ for $t>\tau^{v}_{i}$,

where $(c)$ is due to Lemma 2. Thus, $(b)$ in (28) upper bounded as

Now, we upper bound the random variable in $(a)$ in (28) for all realizations. Consider a new system in which each node $v$ stops playing sub-optimal arm $i$ for $t>\tau^{v}_{i}$. By using Lemma 3, we can calculate an upper bound on $\sum\limits_{v=1}^{m}T^{v}_{i}(\tau^{v}_{i})$. It is easy to see that the same upper bound also holds for $(a)$ in (28). Hence,

Combining (28), (29) and (30) establishes the desired result. ∎

Consider a network $G=(V,E)$ learning a $K$-arm stochastic MAB problem with mean rewards $\mu_{1}~{}\geq~{}\mu_{2}~{}\geq~{}\dots~{}\mu_{K}$. Assume that, each arm distribution is discrete and it assigns a non-zero probability to each possible value. Then, the UCB-user policy followed by any user $v$ in $G$ to learn the above MAB problem is non-altruistic and individually consistent (NAIC) policy.

First, we prove the non-altruistic part. Lemma 2 gives an upper bound on the probability that a node $v$ following the UCB-user policy plays any sub-optimal arm $i$ in round $t$, after it has obtained $l_{i}=\frac{8\ln n}{\Delta_{i}^{2}}$ samples of the arm $i,$ where $n$ is the time horizon. We can treat $\frac{8}{\Delta_{i}^{2}}$ as $a$ in the definition of non-altruistic policy. Observe that, in (29), we upper bounded the expected number of times a node $v$ chooses any sub-optimal arm $i$, after it has access to $l_{i}$ samples of arm $i$, till $n$. Note that, this upper bound is a constant. Hence, the UCB-user policy satisfies the non-altruistic property. Now, we prove the individually consistent part. Recall that, $\omega_{\bar{v}}$ contains actions and the corresponding rewards of the nodes outside the neighbourhood of node $v$, from round $1$ to $n$. Note that, the event $A^{v}_{i}(t)$ defined in the proof of Lemma 2 is independent of any $\omega_{\bar{v}}$, given the event $\{m^{v}_{i}(t)=a,m^{v}_{*}(t)=b\}$. Hence, on the lines of Lemma 2, for $\beta\in(0,1)$, $t>\tau^{v}_{i}$ (same as defined in Lemma 2),

Therefore, the UCB-user policy followed by a node $v$ satisfy individually consistent property, which completes the proof. ∎

Appendix B

Follows from Theorem 2 in , by considering $m_{i}^{G}(n)$ instead of $T_{i}(n)$ in the event $C_{n}$ defined in the respective proof. ∎

Appendix C

Proof of Theorem 3. We now prove $(i)$ in Theorem 3, in the following lemma. With the aid of this lemma, we then prove the second part of the theorem.

Consider a node $v$ in a network $G$. Assume that node $v$ follows an NAIC policy, and suppose [A1] holds. Further, assume that each arm is associated with a discrete distribution such that it assigns a non-zero positive probability to each possible value. Then, for any $\boldsymbol{\theta}\in\boldsymbol{\Theta_{j}}$, and for any $\omega_{\bar{v}}$, the following holds:

Without loss of generality, assume that $\theta_{1}=\theta^{*}$ and $j=2\Rightarrow\boldsymbol{\theta}\in\boldsymbol{\Theta_{2}}$. Consider a new parameter vector $\boldsymbol{\gamma}=\left(\theta_{1},\lambda,\theta_{3},\dots,\theta_{K}\right)$ such that $\mu(\lambda)>\mu(\theta^{*})$, $j\neq 1$. Note that, arm 1 is optimal under parameter vector $\boldsymbol{\theta}$, while arm 2 is optimal under parameter vector $\boldsymbol{\gamma}$. Let $X_{2,1},\dots,X_{2,n}$ be $n$ i.i.d samples generated from the sub-optimal arm 2’s distribution with parameter vector $\boldsymbol{\theta}$. Define

For any $v\in V$ and any sub-optimal arm $j$, and $0<a<\delta$, we define

where $\hat{kl}_{m_{2}^{v}(n)}=\sum\limits_{u\in\mathcal{N}(v)}\sum\limits_{t=1}^{T_{2}^{u}(n)}\ln\left(\frac{f(X_{2,t}^{u};\theta_{2})}{f(X_{2,t}^{u};\lambda)}\right)$, since $\{X^{u}_{2,t}\}_{u\in\mathcal{N}(v)}$ are i.i.d. For convenience, let $g_{n}=\frac{(1-\delta)\ln n}{kl(\theta_{2}||\lambda)}$ and $h_{n}=(1-a)\ln n$. For a given $\omega_{\bar{v}}$, observe that $C_{n}^{v}$ is a disjoint union of events of the form $\{m^{v}_{1}(n)=n_{1},m^{v}_{2}(n)=n_{2},\dots,m^{v}_{K}(n)=n_{K},\hat{kl}_{n_{2}}\leq h_{n}\}$ with $n_{1}+n_{2}\dots+n_{K}=n|\mathcal{N}(V)|$ and $n_{2}\leq g_{n}$. Further, $\{m_{2}^{v}(n)=n_{2}\}$ is also a disjoint union of the events of the form $\{\cap_{u\in\mathcal{N}(v)}T^{u}_{2}(n)=q_{u}\}$ with $\sum\limits_{u\in\mathcal{N}(v)}q_{u}=n_{2}$. Since $\boldsymbol{\gamma}=(\theta_{1},\lambda,\theta_{3},\dots,\theta_{K})$ and $\boldsymbol{\theta}=(\theta_{1},\theta_{2},\theta_{3},\dots,\theta_{K})$, we write

However, $\prod\limits_{u\in\mathcal{N}(v)}\prod\limits_{t=1}^{q_{u}}\frac{f(X_{2,t}^{u};\lambda)}{f(X_{2,t}^{u};\theta_{2})}=\exp(-\hat{kl}_{n_{2}})$. Therefore,

Note that, $\exp(-\hat{kl}_{n_{2}})\geq n^{-(1-a)}$, since $\hat{kl}_{n_{2}}\leq h_{n}$ in the region of integration. Therefore,

Note that, $n|\mathcal{N}(v)|-m^{v}_{2}(n)$ is a non-negative random variable and $kl(\theta_{2}||\lambda)>0$. Therefore, applying Markov’s inequality to the right-hand side in the above equation, we obtain

for $0<a<\delta$, since arm 2 is the unique optimal arm under $\gamma$. Hence,

due to $\frac{1-a}{1-\delta}>1$ and the maximal version of the Strong Law of Large Numbers which is given below.

Maximal version of SLLN : Let $\{X_{t}\}$ be a sequence of independent real-valued random variables with positive mean $\mu>0$. Then,

Part (iii) of assumption, $[A1]$, guarantees the existence of a $\lambda\in\Theta$ such that $\mu(\theta_{1})<\mu(\lambda)<\mu(\theta_{1})+\delta$ holds. Combining $\mu(\theta_{1})>\mu(\theta_{2})$ with the part (i) of $[A1]$, we obtain $0<kl(\theta_{2}||\theta_{1})<\infty$. From part (ii) of $[A1]$, we deduce that $|kl(\theta_{2}||\theta_{1})-kl(\theta_{2}||\lambda)|<\epsilon$, since $\mu(\theta_{1})\leq\mu(\lambda)\leq\mu(\theta_{1})+\delta$ for some $\delta$. Let $\epsilon$ be $\delta kl(\theta_{2}||\theta_{1})$. Hence, we write the following:

Hence, we have proved that for any $v\in V$, $\omega_{\bar{v}}$ and any sub-optimal arm $j$,

which completes the proof of this lemma, and establishes $(i)$ in Theorem 3. ∎

With the help of this, we now prove the second part of Theorem 3.

Note that, the notation in (40) is same as used in Theorem 1, Lemma 3. Let $L_{G}\left(\frac{1-\delta}{1+\delta}\right)\frac{\log n}{kl(\theta_{j}||\theta_{1})}$ be the solution of (40). Thus,

Appendix D

Without loss of generality we consider that node 1 is the center node and node 2 through $m_{n}$ are leaf nodes. Since a policy does not possess any information in the first round, it chooses arm 1 with probability $p_{1}$ and arm 2 with probability $p_{2}$, such that $0\leq p_{1},p_{2}\leq 1$ and $p_{1}+p_{2}=1$. Now, we find the expected number of nodes that chose the arm with parameter $\mu_{b}$ in the first round as follows:

since MAB is $(\mu_{a},\mu_{b})$ with probability $\frac{1}{2}$, and is $(\mu_{b},\mu_{a})$ with probability $\frac{1}{2}$. Henceforth, for convenience, we replace $a$ with 1 and $b$ with 2. Let $m^{G,v}_{i}(t)$ be a random variable indicating the total number of times arm $i$ has been chosen by node $v$ and its one hop neighbours till round $t$, in the network $G$. Note that, $m^{G_{n}}_{2}(1)$ is equals to $m^{G_{n},1}_{2}(1)$, since the network in consideration is a star network with node 1 as the center node. Therefore,

Appendix E

Without loss of generality, we assume that node 1 is the center node in the star network. Under FYL policy, for $2\leq u\leq m$, $a^{u}(t)=a^{1}(t-1)$ for $t>1$. Hence, for any sub-optimal arm $i$,

since $T_{i}^{1}(n-1)\leq T_{i}^{1}(n)$. Now, we find an upper bound on $T_{i}^{1}(n)$ under FYL policy. Let $\tau_{1}$ be the least time step at which $m^{1}_{i}(\tau_{1})$ is atleast $l_{i}=\frac{8\ln n}{\Delta_{i}^{2}}$. Observe that, under FYL policy $T_{i}^{1}(\tau_{1})=\lceil\frac{l_{i}}{m}\rceil$. Since, the center node has chosen arm $i$ for $\lceil\frac{l_{i}}{m}\rceil$ times, $(m-1)$ leaf nodes must have also selected arm $i$ for the same number of times. This leads to $m^{1}_{i}(\tau_{1})=l_{i}$. Let $B_{i}^{1}(t)$ be the event that node-1 chooses arm $i$ in round $t$. Hence,

By using the analysis in Theorem 1, we obtain

where we have substituted $l_{i}=\frac{8\ln n}{\Delta_{i}^{2}}$. Therefore, the expected regret of the FYL policy on an $m$-node star network upto $n$ number of rounds is upper bounded as:

Appendix F

Since the leader node (a node in the given dominating set) in a particular component uses samples only from its neighbours in the same component, we can upper bound the expected regret of each component using Theorem 5. We get the desired result by adding the expected regrets of all the components. ∎