Performance and Convergence of Multi-user Online Learning

Introduction

In this paper we study the dynamic spectrum access and spectrum sharing problem in a learning context. Specifically, we consider a set of $N$ common channels shared by a set of $M$ users. A channel has time varying rate $r(t)$, and its statistics are not completely known by the users. Thus each user needs to employ some type of learning to figure out which channels are of better quality, e.g., in terms of their average achievable rates. At the same time, simultaneous use of the same channel by multiple users will result in reduced rate due to interference or collision. The precise form of this performance degradation may or may not be known to the user. Thus the users also need to use learning to avoid excess interference or congestion. Furthermore, each user may have private information that is not shared, e.g., users may perceive channel quality differently due to difference in location as well as individual modulation/coding schemes.

Without a central agent, and in the presence of information decentralization described above, we are interested in the following questions: (1) for a given common learning algorithm, does the multiuser learning process converge, and (2) if it does, what is the quality of the equilibrium point with respect to a globally optimal spectrum allocation scheme, one that could be computed for a global objective function with full knowledge of channel statistics as well as the users’ private information.

A few recent studies have addressed these questions in some special cases. For instance, in it was shown that learning using a sample-mean based index policy leads to a socially optimal (sum of individual utilities) allocation when channels evolve as iid processes and colliding players get zero reward provided that this optimal allocation is such that each user occupies one of the $M$ best channels (in terms of average rates). This precludes the possibility that not all users may have the same set of $M$ best channels, and that in some cases the best option is for multiple users to share a common channel, e.g., when $N<M$.

In this study we investigate under what conditions the socially optimal allocation is achievable by considering different levels of communication (or cooperation) allowed among users, and different levels of uncertainty on the channel statistics. The general conclusion, as intuition would suggest, is that when the cooperation of users increases and the channel uncertainty decreases it becomes easier to achieve the socially optimal welfare. Specifically, we assume that the rate (or reward) user $i$ gets from channel $j$ at time $t$ is of the form $r_{j}(t)g_{j}(n_{j}(t))$ where $r_{j}(t)$ is the rate of channel $j$ at time $t$, $n_{j}(t)$ is the number of users using channel $j$ at time $t$, and $g_{j}$ is the user independent interference function (IF) for channel $j$. This model is richer than the previously used models since $r_{j}(t)$ can represent environmental effects such as fading or primary user activity, while $g_{j}$ captures interactions between users. We consider the following three cases.

In the first case (C1), each channel evolves as an iid random process in time, the users do not know the channel statistics, nor the form of the interference, nor the total number of users present in the system, and no direct communication is allowed among users. A user can measure the overall rate it gets from using a channel but cannot tell how much of it is due to the dynamically changing channel quality (i.e., what it would get if it were the only user) vs. interference from other users. In this case, we show that if all users follow the Exp3 algorithm then the channel allocation converges to a set of pure Nash equilibria (PNE) of a congestion game defined by the IFs and mean channel rates. In this case a socially optimal allocation cannot be ensured, as the set of PNE are of different quality, and in some cases the socially optimal allocation may not be a PNE.

In the second case (C2), each channel again evolves as an iid random process in time, whose statistics are unknown to the user. However, the users now know the total number of users in the system, as well as the fact that the quantitative impact of interference is common to all users (i.e., user independent), though the actual form of the interference function is unknown. In other words the rate of channel $j$ at time $t$ is perceived by user $i$ as $h_{j}(t,n_{j}(t))$ so user $i$ cannot distinguish between components $r_{j}(t)$ and $g_{j}(n_{j}(t))$. Furthermore, users are now allowed minimal amount of communication when they happen to be in the same channel, specifically to find out the total number of simultaneous users of that channel. In this case we present a sample-mean based randomized learning policy that achieves socially optimal allocation as time goes to infinity, with a sub-linear regret over the time horizon with respect to the socially optimal allocation.

In the third case (C3), as in case (C2) the users know the total number of users in the system, as well as the fact that the IF is user independent and decreasing without knowing the actual form of the IF. However, the channels are assumed to have constant, albeit unknown, rates. We show that even without any communication among users, there is a randomized learning algorithm that achieves the socially optimal allocation in finite time.

It’s worth pointing out that in the settings outlined above, the users are non-strategic, i.e., each user simply follow a pre-set learning rule rather than playing a game. In this context it is reasonable to introduce minimal amount of communication among users and assume they may cooperate. It is possible that even in this case the users may not know their IF but only the total rate they get for lack of better detecting capabilities (e.g., they may only be able to detect the total received SNR as a result of channel rate and user interference).

Online learning by a single user was studied by , in which sample-mean based index policies were shown to achieve logarithmic regret with respect to the best single-action policy without a priori knowledge of the statistics, and are order-optimal, when the rewards are given by an iid process. In Markovian rewards are considered, with focusing on restless reward processes, where a process continues to evolve according to a Markov chain regardless of the users’ actions. In all these studies learning algorithms were developed to achieve logarithmic regret. Multi-user learning with iid reward processes have been studied in a dynamic spectrum context by , with a combinatorial structure adopted in , and with collision and random access models in . In , convergence of multi-user learning with Exp3 algorithm to pure Nash equilibrium is investigated under the collision and fair sharing models. In the collision model, when there is more than one user on a channel all get zero reward, whereas in the random access model one of them, selected randomly, gets all the reward while others get zero reward. In the fair sharing model, a user’s utility is inversely proportional to the number of users who are on the same channel with the user. Note that these models do not capture more sophisticated communication schemes where the rate a user gets is a function of the received SNR of the form $g_{j}(n)=f_{j}(\frac{P_{t}}{N_{0}+(n-1)P_{t}})=$ where $P_{t}$ is the nominal transmit power of all users and $N_{0}$ the noise. Moreover, in the above studies the socially optimal allocation is a rather simple one: it is the orthogonal allocation of users to the first $M$ channels with the highest mean rewards. By contrast, we model a more general interference relationship among users, in which an allocation with users sharing the same channel may be the socially optimal one. The socially optimal allocations is not trivial in this case and additional mechanisms may be needed for the learning algorithms to converge.

All of the above mentioned work assumes some level of communication between the users either at the beginning or during the learning. If we assume no communication between the users, achieving the socially optimal allocation seems very challenging in general. Then one may ask if it is possible to achieve some kind of equilibrium allocation. Kleinberg et. al. showed that it is possible for the case when the channel rates are constant and the users do not know the IFs. They show that when the users use aggregate monotonic selection dynamics, a variant of Hedge algorithm , the allocation converges to weakly stable equilibria which is a subset of Nash equilibria (NE) of the congestion game defined by the IFs. They show that for almost all congestion games weakly stable equilibria is the same as PNE.

Other than the work described above considers spatial congestion games, a generalization of congestion games and gives conditions under which there exists a PNE and best-response play converges to PNE. A mechanism design approach for socially optimal power allocation when users are strategic is considered in .

The organization of the remainder of this paper is as follows. In Sect. 2 we present the notations and definitions that will be used throughout the paper. In Sects. 3, 4, 5 we analyze the cases stated in (C1), (C2), (C3) and derive the results respectively. Conclusion and future research is given in Sect. 6.

Preliminaries

Let $K_{j}(\sigma)$ be the set of users on channel $j$ when the action profile is $\sigma$. Let $\mathcal{A}^{*}=\arg\max_{\sigma\in\mathcal{S}}\sum_{i=1}^{M}\mu_{\sigma_{i}}g_{\sigma_{i}}(K_{\sigma_{i}}(\sigma))=\arg\max_{\sigma\in\mathcal{S}}\sum_{j=1}^{N}\mu_{j}K_{j}(\sigma)g_{j}(K_{j}(\sigma))$ be the set of socially optimal allocations and denote by $\sigma^{*}$ any action profile that is in the set $\mathcal{A}^{*}$. Let $v^{*}$ denote the socially optimal welfare, i.e., $v^{*}=\sum_{i=1}^{M}\mu_{\sigma^{*}_{i}}g_{\sigma^{*}_{i}}(K_{\sigma^{*}_{i}}(\sigma^{*}))$ and $v^{*}_{j}$ denote the payoff a user gets from channel $j$ under the socially optimal allocation, i.e., $v^{*}_{j}=\mu_{j}g_{j}(K_{j}(\sigma^{*}))$ if $K_{j}(\sigma^{*})\neq 0$. Note that any permutation of actions in $\sigma^{*}$ is also a socially optimal allocation since IFs are user-independent.

For any policy $\pi$, the regret at time $n$ is

where expectation is taken with respect to the random nature of the rates and the randomization of the policy. Note that for a deterministic policy expectation is only taken with respect to the random nature of the rates. For any randomized policy $\pi_{i}$, let $p_{i}(t)=(p_{i1}(t),p_{i2}(t),\ldots,p_{iN}(t))$ be the mixed strategy of user $i$ at time $t$, i.e., a probability distribution on $\{1,2,\ldots,N\}$. For a profile of policies $\pi=[\pi_{1},\pi_{2},\ldots,\pi_{M}]$ for the users let $p(t)=(p_{1}(t)^{T},p_{2}(t)^{T},\ldots p_{M}(t)^{T})^{T}$ be the profile of mixed strategies at time $t$, where $p_{i}(t)^{T}$ is the transpose of $p_{i}(t)$. Then $\sigma_{i}(t)$ is the action sampled from the probability distribution $p_{i}(t)$. The dependence of $p$ to $\pi$ is trivial and not shown in the notation.

Allocations Achievable with Exp3 Algorithm (Case 1)

In this section we relate the strategy update rule of Exp3 under assumptions (C1) to a congestion game. Exp3 as given in Fig. 1 is a randomized algorithm consisting of an exploration parameter $\gamma$ and weights $w_{ij}$ that depend exponentially on the past observations where $i$ denotes the user and $j$ denotes the channel. Each user runs Exp3 independently but we explicitly note the user dependence because a user’s action affects other users’ updates.

At any time step before the channel rate and user actions are drawn from the corresponding distributions, let $R_{j}$ denote the random variable corresponding to the reward of the $j$th channel. Let $G_{ij}=g_{j}(1+K^{\prime}_{j}(i))$ be the random variable representing the payoff user $i$ gets from channel $j$ where $K^{\prime}_{j}(i)$ is the random variable representing the number of users on channel $j$ other than user $i$. Let $U_{ij}=R_{j}G_{ij}$ and $\bar{u}_{ij}=E_{j}[E_{-i}[U_{ij}]]$ be the expected payoff to user $i$ by using channel $j$ where $E_{-i}$ represents the expectation taken with respect to the randomization of players other than $i$, $E_{j}$ represents the expectation taken with respect to the randomization of the rate of channel $j$. Since the channel rate is independent of users’ actions $\bar{u}_{ij}=\mu_{j}\bar{g}_{ij}$ where $\bar{g}_{ij}=E_{-i}[G_{ij}]$.

Under (C1) when all players use Exp3, the derivative of the continuous-time limit of Exp3 is the replicator equation given by

We consider the effect of user $i$’s action $\sigma_{i}(t)$ on his probability update on channel $j$. We have two cases: $\sigma_{i}(t)=j$ and $\sigma_{i}(t)\neq j$. Let $A_{i,j}^{\gamma,t}=\exp\left(\frac{\gamma U_{ij}(t)}{p_{ij}(t)N}\right)$.

The continuous time process is obtained by taking the limit $\gamma\rightarrow 0$, i.e., the rate of change in $p_{ij}$ with respect to $\gamma$ as $\gamma\rightarrow 0$. Then, dropping the discrete time script $t$,

Consider the case $\sigma_{i}(t)=k\neq j$. Then,

Then from (3) and (4), the expected change in $p_{ij}$ with respect to the probability distribution $p_{i}$ of user $i$ over the channels is

Taking the expectation with respect to the randomization of channel rates and other users’ actions we have

Lemma 1 shows that the dynamics of a user’s probability distribution over the actions is given by a replicator equation which is commonly studied in evolutionary game theory . With this lemma we can establish the following theorem.

For all but a measure zero subset of $^{2N}$ from which the $\mu_{j}$’s and $g_{j}$’s are selected, when $\gamma$ in Exp3 is arbitrarily small, the action profile converges to the set of PNE of the congestion game $({\cal M},{\cal N},({\cal S}_{i})_{i\in{\cal M}},(\mu_{j}g_{j})_{j\in{\cal N}})$.

Because the replicator equation in Lemma 1 is identical to the replicator equation in , the proof of converge to PNE follows from . Here, we briefly explain the steps in the proof. Defining the expected potential function to be the expected value of the potential function $\phi$ where expectation is taken with respect to the user’s randomization one can show that the solutions of the replicator equation converges to the set of fixed points. Then the stability analysis using the Jacobian matrix yields that every stable fixed point corresponds to a Nash equilibrium. Then one can prove that for any stable fixed point the eigenvalues of the Jacobian must be zero. This implies that every stable fixed point corresponds to a weakly stable Nash equilibrium strategy in the game theoretic sense. Then using tools from algebraic geometry one can show that almost every weakly stable Nash equilibrium is a pure Nash equilibrium of the congestion game.

We also need to investigate the error introduced by treating the discrete time update rule as a continuous time process. However, by taking $\gamma$ infinitesimal we can approximate the discrete time process by the continuous time process. For a discussion when $\gamma$ is not infinitesimal one can define approximately stable equilibria . ∎

The main difference between Exp3 and Hedge is that in Exp3 users do not need to observe the payoffs from the channels that they do not select, whereas Hedge assumes complete observation. In addition to that, we considered the dynamic channel rates which is not considered in .

An Algorithm for Socially Optimal Allocation with Sub-linear Regret (Case 2)

The following will be useful in the proof of the main theorem of this section.

Let $X_{i},i=1,2,\ldots$ be a sequence of independent Bernoulli random variables such that $X_{i}$ has mean $q_{i}$ with $0\leq q_{i}\leq 1$. Let $\bar{X}_{k}=\frac{1}{k}\sum_{i=1}^{k}X_{i}$ , $\bar{q}_{k}=\frac{1}{k}\sum_{i=1}^{k}q_{i}$. Then for any constant $\epsilon\geq 0$ and any integer $n\geq 0$,

When all players use RLA the regret with respect to the socially optimal allocation is $O(n^{\frac{2M-1+2\gamma}{2M}})$ where $\gamma$ can be arbitrarily small.

Let $H(t)$ be the event that at time $t$ there exists at least one user that computed the socially optimal allocation incorrectly. Let $\omega$ be a sample path. Then

Let $\epsilon^{i}_{j,k}(t)=\sqrt{\frac{a\ln t}{T^{i}_{j,k}(t)}}$. Then $T^{i}_{j,k}(t)\geq\frac{a\ln t}{\epsilon^{2}}\Rightarrow\epsilon\geq\sqrt{\frac{a\ln t}{T^{i}_{j,k}(t)}}=\epsilon^{i}_{j,k}(t)$. Therefore,

where (10) follows from the Chernoff-Hoeffding inequality.

Now we will bound $P\left(T^{i}_{j,l}(t)<\frac{a\ln t}{\epsilon^{2}}\right)$. Let $TR^{i}_{j,l}(t)$ be the number of time steps in which player $i$ played channel $j$ and observed $l$ users on channel $j$ in the time steps where all players randomized up to time $t$. Then

Now we define new Bernoulli random variables $X^{i}_{j,l}(s)$ as follows: $X^{i}_{j,l}(s)=1$ if all players randomize at time $s$ and player $i$ selects channel $j$ and observes $l$ players on it according to the random draw. $X^{i}_{j,l}(s)=0$ else. Then $TR^{i}_{j,l}(t)=\sum_{s=1}^{t}X^{i}_{j,l}(s)$. $P(X^{i}_{j,l}(s)=1)=\rho_{s}p_{l}$ where $p_{l}=\frac{{M-1\choose l-1}{M+N-l-2\choose N-2}}{{M+N-1\choose N-1}}$ and $\rho_{s}=\frac{1}{s^{(1/2)-\gamma}}$. Let $s_{t}=\sum_{s=1}^{t}\frac{1}{s^{(1/2)-\gamma}}$ Then

where (13) follows from Lemma 3. Let $\tau(M,N,\epsilon,\gamma,\gamma^{\prime},a)$ be the time that for all $k\in\{1,2,\ldots,M\}$.

where $0<\gamma^{\prime}<\gamma$. Then for all $t\geq\tau(M,N,\epsilon,\gamma,\gamma^{\prime},a)$ (14) will hold since RHS increases faster than LHS. Thus we have for $t\geq\tau(M,N,\epsilon,\gamma,\gamma^{\prime},a)$

Let $a=1$. Then continuing from (9) by substituting (10) and (15) we have

Thus we proved that the expected number of time steps in which there exists at least one user that computed the socially optimal allocation incorrectly is finite. Note that because RLA explores with probability $\frac{1}{t^{1/2M-\gamma/M}}$, the expected number of time steps in which all the players are not randomizing up to time $n$ is

Note that players can choose $\gamma$ arbitrarily small, increasing the finite regret due to $\tau(M,N,\epsilon,\gamma,\gamma^{\prime},1)$. Thus if we are interested in the asymptotic performance then $\gamma>0$ can be arbitrarily small.

Now we do the worst case analysis. We classify the time steps into two. Good time steps in which all the players know the socially optimal allocation correctly and none of the players randomize excluding the randomizations done for settling down to the socially optimal allocation. Bad time steps in which there exists a player that does not know the socially optimal allocation correctly or there is a player that randomizes excluding the randomizations done for settling down to the socially optimal allocation. The number of Bad time steps in which there exists a player that does not know the socially optimal allocation correctly is finite while the number of time steps in which there is a player that randomizes excluding the randomizations done for settling down to the socially optimal allocation is $O(n^{\frac{2M-1+2\gamma}{2M}})$. The worst case is when each bad step is followed by a good step. Then from this good step the expected number of times to settle down to the socially optimal allocation is $\left(1-\frac{1}{{M+z^{*}-1\choose z^{*}-1}}\right)/\left(\frac{1}{{M+z^{*}-1\choose z^{*}-1}}\right)$ where $z^{*}$ is the number of channels which has at least one user in the socially optimal allocation. Assuming in the worst case the sum of the utilities of the players is when they are not playing the socially optimal allocation we have

Note that we mentioned earlier, under a classical multi-armed bandit problem approach as cited before , a logarithmic regret $O(\log n)$ is achievable. The fundamental difference between these studies and the problem in the present paper is the following: Assume that at time $t$ user $i$ selects channel $j$. This means that $i$ selects to observe an arm from the set $\{(j,k):k\in{\cal M}\}$ but the arm assigned to $i$ is selected from this set depending on the choices of other players.

Also note that in RLA a user computes the socially optimal allocation according to its estimates at each time step. This could pose significant computational effort since integer programming is NP-hard in general. However, by exploiting the stability condition on the socially optimal allocation a user may reduce the number of computations; this is a subject of future research.

An Algorithm for Socially Optimal Allocation (Case 3)

In this section we assume that $g_{j}(n)$ is decreasing in $n$ for all $j\in{\cal N}$. For simplicity we assume that the socially optimal allocation is unique up to the permutations of $\sigma^{*}$. When this uniqueness assumption does not hold we need a more complicated algorithm to achieve the socially optimal allocation. All users use the Random Selection (RS) algorithm defined in Fig. 3. RS consists of two phases. Phase 1 is the learning phase where the user randomizes to learn the interference functions. Let $B_{j}(t)$ be the set of distinct payoffs observed from channel $j$ up to time $t$. Then the payoffs in set $B_{j}(t)$ can be ordered in a decreasing way with the associated indices $\{1,2,\ldots,|B_{j}(t)|\}$. Let $O(B_{j}(t))$ denote this ordering. Since the IFs are decreasing, at the time $|B_{j}(t)|=M$, the user has learned $g_{j}$. At the time $|\cup_{j=1}^{N}B_{j}(t)|=MN$, the user has learned all IFs. Then, the user computes $\mathcal{A}^{*}$ and phase 2 of RS starts where the user randomizes to converge to the socially optimal allocation.

Under the assumptions of (C3) if all players use RS algorithm to choose their actions, then the expected time to converge to the socially optimal allocation is finite.

Let $T_{OPT}$ denote the time the socially optimal allocation is achieved, $T_{L}$ be the time when all users learn all the IFs, $T_{F}$ be the time it takes to reach the socially optimal allocation after all users learn all the IFs. Then $T_{OPT}=T_{L}+T_{F}$ and $E[T_{OPT}]=E[T_{L}]+E[T_{F}]$. We will bound $E[T_{L}]$ and $E[T_{F}]$. Let $T_{i}$ be the first time that $i$ users have learned the IFs. Let $\tau_{i}=T_{i}-T_{i-1},i=1,2,\ldots,M$ and $T_{0}=0$. Then $T_{L}=\tau_{1}+\ldots+\tau_{M}$. Define a Markov chain over all $N^{M}$ possible configurations of $M$ users over $N$ channels based on the randomization of the algorithm. This Markov chain has a time dependent stochastic matrix which changes at times $T_{1},T_{2},\ldots,T_{M}$. Let $P_{T_{0}},P_{T_{1}},\ldots,P_{T_{M}}$ denote the stochastic matrices after the times $T_{0},T_{1},\ldots,T_{M}$ respectively. This Markov chain is irreducible at all times up to $T_{M}$ and is reducible with absorbing states corresponding to the socially optimal allocations after $T_{M}$. Let $\hat{T}_{1},\hat{T}_{2},\ldots\hat{T}_{M}$ be the times that all configurations are visited when the Markov chain has stochastic matrices $P_{T_{0}},P_{T_{1}},\ldots,P_{T_{M-1}}$ respectively. Then because of irreducibility and finite states $E[\hat{T}_{i}]<z_{1},i=1,\ldots,M$ for some constant $z_{1}>0$ . Since $\tau_{i}\leq\hat{T}_{i},i=1,\ldots,M$ a.s. we have $E[T_{L}]<Mz_{1}$. For the Markov chain with stochastic matrix $P_{T_{M}}$ all the configurations that do not correspond to the socially optimal allocation are transient states. Since starting from any transient state the mean time to absorption is finite $E[T_{F}]<z_{2}$, for some constant $z_{2}>0$. ∎

Conclusion

In this paper we studied the decentralized multiuser resource allocation problem with various levels of communication and cooperation between the users. Under three different scenarios we proposed three algorithms with reasonable performance. Our future reserach will include characterization of achievable performance regions for these scenarios. For example, in case 2 we are interested in finding an optimal algorithm and a lower bound on the performance.

References