Degenerate Feedback Loops in Recommender Systems

Introduction

Recommender systems are increasingly used to provide users with personalized product and information offerings (Ben Schafer, Konstan, and Riedl, 2001; Lu et al., 2015; Covington, Adams, and Sargin, 2016). These systems employ user’s personal characteristics and past behaviors to generate a list of items that are individually tailored to the user’s preferences. Whilst extremely successful commercially, there are growing concerns that such systems might lead to a self-reinforcing pattern of narrowing exposure and shift in user’s interest, problems that are often referred to in the literature as “echo chamber” and “filter bubble”. A significant amount of research has therefore been devoted to deriving ways to favor diversity in the set of items an individual may be exposed to (see Kunaver and Porl (2017) for a review). However, current understanding of the echo chamber and filter bubble effects is limited and experimental analysis reports conflicting results.

In this paper, we define as echo chamber the effect of a user’s interest being positively or negatively reinforced by repeated exposure to a certain item or category of items, thereby generalizing the definition in Sunstein (2009), where the term is used to refer to over- and limited-exposure to similar political opinions reinforcing one’s existing beliefs. We focus the definition of filter bubble introduced by Pariser (2011) to describe just the fact that recommender systems select limited content to serve users online. We provide a theoretical treatment that allows us to consider the echo chamber and filter bubble effects separately. We view user’s interest as a dynamical system and treat interest extremes as degeneracy points of the system. We consider different models of dynamics and identify sets of sufficient conditions that make them degenerate over time. We then use this analysis to understand the role played by the recommender system. Finally, we showcase the interplay between the user’s dynamics and the recommender system actions in a simulation study using synthetic data and several classic bandit algorithms. The results reveal several pitfalls of recommender system design and point towards mitigation strategies.

Related Work

Through an analysis on the MovieLens dataset, Nguyen et al. (2014) found that the diversity of items recommended, and those users engage with, gets narrower over time. The paper asks whether there is a “natural” tendency of degeneration in user interest. Our paper takes steps toward answering this question by providing theoretical conditions for user interest degeneracy.

In the social sciences literature, Flaxman, Goel, and Rao (2016) found that online services are associated with increased political polarization between users as well as increased exposure to the less preferred side of political opinions. Their seemingly counter-intuitive findings are not contradictory according to our results: systems with some level of random exploration can be degenerative. Barberá et al. (2015) also presented evidence of echo chamber related to political issues on Twitter. On the other hand, Borgesius et al. (2016); Beam, Hutchens, and Hmielowski (2018); Nechushtai and Lewis (2018) found counter-evidence on online news consumption. Another work by Bakshy, Messing, and Adamic (2015) measured the effect of user choices separately from that of the recommendation algorithm, and found that individual choices play a larger role than the algorithm in creating echo chamber on Facebook. This supports our viewpoint that user interests degenerate or not depending on their internal dynamics, the recommender system can only slow down or accelerate the process of degeneration.

Model

Given a recommendation $a_{t}=(a^{1}_{t},\ldots,a^{l}_{t})\in\mathcal{M}^{l}$, the user provides some feedback $c_{t}$ based on her current interests $\mu_{t}(a^{1}_{t}),\ldots,\mu_{t}(a^{l}_{t})$. This interaction has multiple effects: in the traditional literature for recommender systems, the feedback $c_{t}$ is used to update the internal model $\theta_{t}$ of the recommender system that has been used to obtain the recommendation $a_{t}$, and the new model $\theta_{t+1}$ may depend on $\theta_{t}$, $a_{t}$, and $c_{t}$. In practice $\theta_{t}$ usually predicts the distribution of user feedback to determine which items $a_{t}$ should be presented to the user. In this paper we focus on another effect and consider explicitly that the user’s interaction with the recommender system may change her interest in different items for the next interaction, thus the interest $\mu_{t+1}$ may depend on $\mu_{t}$, $a_{t}$, and $c_{t}$. The full model of interaction is depicted in Fig. 1.

We are interested in studying the evolution of the user’s interest. An example of such an evolution is that the interest is reinforced by user interactions with the recommended items, that is, $\mu_{t+1}(a)>\mu_{t}(a)$ if the user clicks on an item $a$ at time step $t$, while $\mu_{t+1}(a)<\mu_{t}(a)$ if $a$ is shown but not clicked (here $c_{t}\in\{0,1\}^{l}$ can be defined as the indicator vector of clicks to the corresponding items).

To analyze the echo chamber or filter bubble effect, we are interested in understanding when the user’s interest changes extremely, which, in our model, translates to $\mu_{t}(a)$ taking values arbitrarily different from the initial interest $\mu_{0}(a)$: large positive values indicate that the user becomes extremely interested in item $a$, while large negative values indicate that the user dislikes $a$. Formally, for a finite item set $\mathcal{M}$, we can ask if the $L^{2}$ norm $\|\mu_{t}-\mu_{0}\|_{2}=\left(\sum_{a\in\mathcal{M}}(\mu_{t}(a)-\mu_{0}(a))^{2}\right)^{1/2}$ can grow arbitrarily large: the user’s interest sequence $\mu_{t}$ is called weakly degenerate if

A stronger notion of degeneracy, which also requires that once $\mu_{t}$ drifted away from $\mu_{0}$ it remains so, is strong degeneracy: the sequence $\mu_{t}$ is strongly degenerate if

In the next section we show that weak or strong degeneracy occurs under mild sufficient conditions on the evolutionary dynamics of $\mu_{t}$.

There are multiple ways to extend the above definitions to the case of an infinite item set $\mathcal{M}$. For simplicity, we only consider here replacing $\|\mu_{t}-\mu_{0}\|_{2}$ with $\sup_{a\in\mathcal{M}}|\mu_{t}(a)-\mu_{0}(a)|$ in Eqs. (1) and (2), which is equivalent to the original definitions when $\mathcal{M}$ is finiteAs such, we could have used $\sup_{a\in\mathcal{M}}|\mu_{t}(a)-\mu_{0}(a)|$ in our original definitions, but we prefer $\|\mu_{t}-\mu_{0}\|_{2}$ as it also provides some information about the “average” deviation of the user’s interest over the different items at any finite time $t$..

User Interest Dynamics – Echo Chamber

As items often represent diverse categories of things, we make the simplifying assumption that they are independent from each other. By setting $l=1$ and $a^{1}_{t}=a$ for all $t$ (i.e., $\mathcal{M}=\{a\}$), we can remove the influence of the recommender system and consider the dynamics of the user’s interest separately. This allows us to analyze the echo chamber effect: what happens to the interest $\mu_{t}(a)$ if item $a$ is served infinitely often (i.o.).

Since $a$ is fixed, to simplify the notation, we write $\mu_{t}$ instead of $\mu_{t}(a)$ in this section. Given $a_{t}$, according to Fig. 1, $\mu_{t+1}$ is a—possibly stochastic—function of $\mu_{t}$ (as $\mu_{t+1}$ depends on $c_{t}$ and $\mu_{t}$, and $c_{t}$ depends on $\mu_{t}$). Below we discuss the general case when the drift $\mu_{t+1}-\mu_{t}$ is a nonlinear stochastic function; deterministic models for the drift are considered in Appendix B.

be the expected increment $\mu_{t+1}-\mu_{t}$ when $\mu_{t}=\mu$. We also define

to be the cumulative distribution of the increment. The asymptotic behavior of $\mu_{t}$ depends on $f$, but under mild assumptions the system degenerates weakly (Theorem 1) or strongly (Theorem 2)The proofs of these theorems are given in Appendix A..

The assumptions guarantee that within any closed bounded interval there is a constant probability that the random walk escapes to the left/right when starting to the left/right of $\mu_{\circ}$ respectively. Under stronger conditions it is possible to guarantee the divergence of the random walk. We state a simple version of the theorem, but note that the result can be generalized in many ways.

Intuitively, weak degeneracy occurs in a stochastic environment if the user’s interest has some non-zero probability of drifting up when above some threshold, and of drifting down when below. Strong degeneracy holds if additionally $|\mu_{t+1}-\mu_{t}|$ is bounded and for $\mu_{t}$ sufficiently large/small the increment $\mu_{t+1}-\mu_{t}$ has positive/negative drift that is larger than a constant.

Theorems 1 and 2 show that the user’s interest degenerates under very mild conditions, in particular, in the model we consider in our simulation studies. Thus, in such cases degeneracy can only be avoided if an item (or item category) is showed only finitely many times; otherwise one can only hope to control how fast $\mu_{t}$ degenerates (i.e. tends to $\infty$).

System Design Role – Filter Bubble

In the previous section we discussed conditions for degeneracy for different user interest dynamics. In this section we examine the other side of the story, the influence of recommender system actions in creating filter bubbles. We typically do not know the dynamics of the user’s interest in the real world. However, we consider the relevant scenario to the echo chamber/filter bubble problem where user’s interest in some items has degenerative dynamics, and examine how to design a recommender system that slows down the degeneracy process. We consider three dimensions, namely model accuracy, amount of exploration, and growing candidate pool.

One common goal of recommender systems designers is to increase the prediction accuracy of the internal model $\theta_{t}$. How does model accuracy coupled with greedy optimal $a_{t}$ affect the speed of degeneration? We examine this question for the extreme case of exact predictions, i.e. $\theta_{t}=\mu_{t}$, we call such a prediction model the oracle model. We argue that under the surfacing assumption explained below, the oracle model coupled with greedily optimal action selection results in the quickest degeneracy.

In order to analyze the problem concretely, we focus on the degenerate linear dynamics model for $\mu_{t}(a)$ for $a\in\mathcal{M}$, i.e. $\mu_{t+1}(a)=(1+k)\mu_{t}(a)+b$. Then we can solve for $\mu_{t}(a)$, obtaining

Surfacing Assumption: Let $[m]=\{1,2,\ldots,m\}$ be the candidate set of size $m$. If a subset of items $\mathcal{S}\subset[m]$ leads to positive degenerate dynamics (i.e. $\mu_{t}(a)\rightarrow+\infty$ for all $a\in\mathcal{S}$), then we assume that there exists a time $\tau>0$ such that, for all $t\geq\tau$, $\mathcal{S}$ takes up the top $|\mathcal{S}|$ items in terms of values of $\mu_{t}$, sorted by the base value of the exponential function, $|1+k(a)|$.

The surfacing assumption makes sure that the quickest degenerating items surface out to the top list given enough time of exposure. It can be generalized to nonlinear stochastic dynamics of $\mu_{t}$ provided that the items from $\mathcal{S}$ have an almost surely stable ordering of degeneracy speed $|\mu_{t}(a)-\mu_{0}(a)|/t$ over time.

Under the general surfacing assumption, after time $\tau$, the quickest way to degeneration is to serve the top $l$ items according to $\mu_{t}$, or $\theta_{t}$ of the oracle model. Even if the assumption is violated to some degree, the oracle model still leads to degeneracy very efficiently by picking the top $l$ items according to $\mu_{t}$ which are likely to receive positive feedback due to high $\mu_{t}$, and therefore increasing $\mu_{t+1}$ and reinforcing the past choices.

In practice the recommender system models are inaccurate. We can think of inaccurate models as the oracle model with different levels of noises added to $\theta_{t}$. We discuss inaccurate models in the next section.

Amount of Exploration.

Consider a type of $\epsilon$-random exploration where $a_{t}$ always picks the top $l$ items out of a finite candidate pool $[m]$ with uniform $\epsilon$ noise on $\theta_{t}$, i.e. according to $\theta^{\prime}_{t}=\theta_{t}+U([-\epsilon,\epsilon])$.

Given the same model sequence $\theta_{t}$, the bigger $\epsilon$ is, usually the slower the system degenerates. However, in practice $\theta_{t}$ is learned from observations, and the random exploration added to an oracle model may in fact accelerate degeneration: random exploration can help reveal the most positively degenerating items over time making the surfacing assumption more likely to be true (we show this phenomenon in the simulation experiments below, Fig. 5). In addition, if user interests have degenerative dynamics, even recommending items uniformly at random leads to degeneration, albeit quite slowly.

How do we then make sure that the recommender system does not make user interests degenerate? One way is to limit the number of times an item for which the user’s interest dynamics is degenerative is served to the user. In practice it is hard to detect which items correspond to degenerative dynamics, however we can generally prevent degeneration if all items are served only a finite number of times, which suggests having an ever growing pool of candidate items.

Growing Candidate Pool ℳℳ\mathcal{M}.

With a growing candidate pool, at every time step an additional set of new items becomes available to be served to the user. Hence the domain of the function $\mu_{t}$ expands as $t$ increases. Adding new items at least linearly often is a necessary condition to avoid possible degeneration, since in a finite or any sublinearly growing candidate pool, by the pigeon hole principle, there must exist at least one item that is served i.o., which is degenerate in the worst case scenario (also under general conditions described e.g. in Theorem 2). However, with an at least linearly growing candidate pool $\mathcal{M}$ the system can potentially impose the maximum number of times any item is served to a user and prevent degeneration.

Simulation Experiments

In this section, we consider a simple degenerative dynamics for $\mu_{t}$ and examine degeneration speed under five different recommender system models. We further demonstrate that adding new items to the candidate pool can be an effective solution against system degeneracy.

We create a simulation for the model of interaction between a recommender system and a user of Fig. 1. Consider a possibly growing candidate pool of items of initial size $m_{0}$ and of size $m_{t}$ at time step $t$. At each time step $t$, a recommender system picks the top $l$ out of the $m_{t}$ items $a_{t}=(a_{t}^{1},\ldots,a_{t}^{l})$ according to the internal model $\theta_{t}$ to serve to a user. The user considers each of the $l$ items independently and chooses to click on a (possibly empty) subset of them, thereby generating a binary vector $c_{t}$ of size $l$ where $c_{t}(a_{t}^{i})$ gives the user feedback on item $a_{t}^{i}$, according to $c_{t}(a_{t}^{i})\sim Bernoulli(\phi(\mu_{t}(a_{t}^{i})))$, where $\phi$ is the sigmoid function $\phi(x)=1/(1+e^{-x})$. The system then updates the model $\theta_{t+1}$ based on the past actions, feedbacks and the current model parameter $\theta_{t}$. We assume that the user’s interest increases/decreases by $\delta(a^{\prime})$ if the item $a^{\prime}$ receives/does not receive a click, i.e.

The internal recommender system model is updated according to following five algorithms:

Random Model: Instead of picking top items, the set of items $a_{t}=(a_{t}^{1},\ldots,a_{t}^{l})$ is sampled from a uniform random distribution over the candidate set $U([m_{t}])$.

Oracle: $\theta_{t+1}(a_{t}^{i})=\mu_{t+1}(a_{t}^{i})$, $\forall i$.

Optimal Oracle: $\theta_{t+1}(a_{t}^{i})=\delta(a_{t}^{i})$, $\forall i$. This model does not pick the highest $l$ items according to $\mu_{t}$ but according to $\delta$. Thus, it always picks the fastest degenerating items, therefore maximizing both the long term user engagement $\sum_{t=0}^{\infty}\|c_{t}\|_{1}$ and the degeneracy speed. For a fixed candidate pool $\mathcal{M}$, this model is equivalent to an Oracle that satisfies the Surfacing Assumption.

Upper Confidence Bound Multi-armed Bandit Algorithm (UCB) (Lai, 1987; Auer, Cesa-Bianchi, and Fischer, 2002; Lattimore and Szepesvári, 2019): We use the version of UCB algorithm in Chapter 8 of Lattimore and Szepesvári (2019), however most UCB algorithms perform similarly to the purpose of this experiment. The algorithm prioritizes serving any item from the candidate set that has never been served before. This treatment includes the initial $m_{0}$ items as well as later whenever new items are added to the candidate pool. At time step $t$, UCB serves $l^{\prime}(0\leq l^{\prime}\leq l)$ previously unserved items and the top $l-l^{\prime}$ items according to values of $\theta_{t}$. Define $f(t)=1+t\log^{2}(t)$ and we use the following model update $\theta_{t+1}(a)=\hat{c}_{t}(a)+\sqrt{2\log f(t)/T_{a}(t)}$, where $\hat{c}_{t}$ is the empirical average of feedbacks on item $a$, i.e. $\hat{c}_{t}(a)=\sum_{0\leq i\leq t,a\in a_{i}}^{t}c_{t}(a)/T_{a}(t)$, and $T_{a}(t)$ is the number of times item $a$ has been served up to time $t$, i.e. $T_{a}(t)=\sum_{0\leq i\leq t,a\in a_{i}}1$.

Thompson Sampling Multi-armed Bandit Algorithm (TS) (Thompson, 1933): We initialize $\alpha_{0}(a)=1,\beta_{0}(a)=1$ for any new item $a$. If $a$ is served at time $t$, we perform the update $\alpha_{t+1}(a)=\alpha_{t}(a)+c_{t}(a),\beta_{t+1}(a)=\beta_{t}(a)+1-c_{t}(a)$. At any time $t$, the internal model $\theta_{t}$ is sampled from the corresponding beta distribution $Beta(\alpha_{t},\beta_{t})$.

We examine the echo chamber and filter bubble effects by running the simulation on a candidate pool of fixed size $m_{t}=m=100$ with time horizon $T=5,000$.

In Fig. 2, we show the degeneration of user interest $\mu_{t}$ (left column) and the serving rate (right column) of every item as each recommender model evolves in time. The serving rate of an item shows how often it is served within the report interval. In order to see the distribution clearly, we sort the items according to the z-values at the report time. Although all models cause user interest degeneration, the degeneration speeds are quite different (Optimal Oracle $>$ Oracle, TS, UCB $>$ Random Model). The Oracle, TS and UCB optimize based on $\mu_{t}$ and so we see a positive degenerative dynamics for $\mu_{t}$. The Optimal Oracle optimizes on the degeneration speed directly and not on $\mu_{t}$ so we see both a positive and negative degeneration in $\mu_{t}$. The Random Model also drifts $\mu_{t}$ in both directions, but at a much slower rate. However, overall except for the Random Model, very quickly both the top items served and the top user interests narrow down to the ($l=$) 5 most positively reinforced items.

Speed of Degeneracy

Next, we compare the degeneracy speed for the five recommender system models on both fixed and growing candidate sets. As the $L^{2}$ distance that measures system degeneracy is asymptotically linear for all five models (see Appendix C), we quantify degeneracy speeds by compare empirically $\|\mu_{t}-\mu_{0}\|_{2}/t$ in finite candidate pools for different experiment setups.

Figure 3 shows the degeneracy speed of five models averaged across 30 runs when we take $m=100$ and evolve the system for $T=5,000$ steps. We see that the Optimal Oracle results in the fastest degeneration by far, followed by the Oracle, TS and UCB. The Random Model offers the slowest degeneracy speed.

In Fig. LABEL:fig:changing_m_T we compare the Optimal Oracle, UCB and TS’ degeneracy speed $\|\mu_{t}-\mu_{0}\|_{2}/t$ up to 5,000 time steps and candidate pool sizes $m=10,10^{2},10^{3},10^{4}$. Apart from the Random model, we see that UCB slows down system degeneracy the most given a large candidate pool since it is forced to explore any unserved item first. A larger candidate pool requires a longer time for exploration for the bandit algorithms. As the candidate pool size grows to 10,000 UCB’s degeneracy speed never peaks up given the time horizon, but will eventually grow given a longer time. TS has higher degeneracy speed due to weaker exploration on new items. The Optimal Oracle accelerates degeneration given a larger pool, as it can pick potentially faster degenerative items than from a smaller pool.

Additionally, in Fig. LABEL:fig:changing_m we plot all five models degeneracy speed for $T=20,000$ against the same changing candidate pool sizes. The degeneracy speed of the Optimal Oracle and the Oracle increases with the size of the candidate set, but that of the and Random Model, UCB, and TS decreases. In practice, having a large candidate pool can be a temporary solution to slow down system degeneration.

The Effect of the Noise Level.

Next we show the influence of internal model inaccuracy on degeneracy speed. We compare the Oracle model with different amounts of uniformly random noises, i.e. the system serves the top $l$ items according to the noisy internal model $\theta^{\prime}_{t}=\theta_{t}+U([-\epsilon,\epsilon])$. The candidate pool has fixed size $m=100$. In Fig. 5, we vary $\epsilon$ from 0 to 10. Counter-intuitively adding noise to Oracle accelerates degeneration since faster degenerative items may be selected by chance than those fixed set of top $l$ items ranked by $\mu_{0}$, and more likely satisfies the Surfacing Assumption. Given $\epsilon>0$, as expected, we see a nice monotonically increasing damping effect on degeneracy speed as the noise level grows.

Growing Candidate Pool.

We extend the definition of degeneracy speed to an infinite candidate pool by computing $\sup_{a\in\mathcal{M}}|\mu_{t}(a)-\mu_{0}(a)|/t$ (see Appendix C for an asymptotic analysis). Since the degeneracy speed may not be asymptotically linear for all five models, we examine directly the sup distance $\sup_{a\in\mathcal{M}}|\mu_{t}(a)-\mu_{0}(a)|$ over 10,000 time steps. To construct growing candidate pools at different growth speed, we define a growth function $m_{t}=\left\lfloor m_{0}+lt^{\eta}\right\rfloor$ by varying the growth parameter$\eta=0$ gives a fixed candidate pool, $0<\eta<1$ gives sub-linear growth, $\eta=1$ gives linear growth. $\eta=0,0.5,1$, where $m_{0}=100$. In Fig. 6 we average the results over 10 independent runs. Both the Oracle and the Optimal Oracle for all growth rates are degenerate. The Random Model stops degeneration at sublinear growth, $\eta=0.5$, so does UCB thanks to forced exploration on previously unserved items, although its trajectory has a small upward tilt. The TS model degenerates at sublinear growth but stops degeneration at linear growth $\eta=1$. For all models, the higher the growth rate $\eta$, the slower they degenerate, if they do at all. Overall when applicable, an ideally linearly growing candidate set and continuous random exploration seem to be good remedies against an adversarial dynamics of $\mu_{t}$ to best prevent degeneracy.

Conclusion

We provided a theoretical analysis of the echo chamber and filter bubble effects for recommender systems. We used the dynamical system framework to model user’s interest and treated interest extremes as degeneracy points of the system. We gave formal definitions of system degeneracy and provided sufficient conditions which make the system degenerate with both deterministic and stochastic dynamics. On the recommender system side, we discussed the influence on degeneracy speed of three independent factors in system design, i.e. model accuracy, amount of exploration, and the growth rate of the candidate pool. An oracle model often leads to quick degeneracy of the system, while continuous exploration and a large candidate pool size can help slow it down. The best remedies against system degeneracy we found are continuous random exploration and growing the candidate pool at least linearly.

Our work has two main limitations. First, since user interests are hidden variables that are not directly observed, a good measure or proxy for user interests is necessary in practice to study degeneration reliably. Second, we assumed that items and users are independent from each other – we will extend the theoretical analysis to the case of possibly mutually dependent items and users in a future work.

Acknowledgments

We would like to thank William Isaac, Michael Mathieu, Krishnamurthy Dvijotham, Timothy Mann and Dilan Gorur, for helpful discussions and advice.

References

Appendix A Proofs

which is a contradiction. In order to prove (4) notice that compactness of $[0,B]$ and $[-B,0]$ and the continuity of $F$ at $(\mu,0)$ for all $\mu$ ensures there exists an $\epsilon>0$ depending only on $B$ such that $1-F(\mu,\epsilon)\geq\epsilon$ for all $\mu\in[0,B]$ and $F(\mu,-\epsilon)\geq\epsilon$ for all $\mu\in[-B,0]$. Hence for $n=1+B/\epsilon$

Let $E_{k}$ be the event that $\mu_{t}\in[-B,B]$ for all $t\in\{nk+1,\ldots,n(k+1)+1\}$. Then

We prove that if $B$ is sufficiently large and $\mu>B$ and $\tau=\min\{t:\mu_{t}\in[-B,B]\}$, then

For $\mu<-B$ the same holds, but with $\mu_{t}$ tending to $-\infty$. To see why this implies the result, notice that the previous theorem shows that $\mu_{t}$ eventually leaves $[-B,B]$ almost surely. Each time this happens there is more than 0.5 probability of divergence and a certainty of either divergence or returning to $[-B,B]$. The conditional Borel-Cantelli theorem concludes the proof. To see why (5) and (6) hold, let

which is a martingale with bounded increments since $|\mu_{t+1}-\mu_{t}|$ are bounded. Now $M_{\tau\wedge n}$ is also a martingale. Then by the strong law of large numbers for martingales,

which implies that either $\tau<\infty$ or $\mu_{n}\to\infty$ almost surely. To see the latter case, suppose $\mu$ is sufficiently large, we compute

since $M_{n}/n\rightarrow 0$ a.s. A similar argument applies when $\mu$ is sufficiently small. Finally, Eq. (5) holds by Azuma’s inequality. ∎

Appendix B More on User Interest Dynamics

Recall that since $a$ is fixed, to simplify the notation, we write $\mu_{t},c_{t}$ instead of $\mu_{t}(a),c_{t}(a)$ in this section. We assume $c_{t}=g(\mu_{t})$ for some deterministic function $g$ and analyze the one-dimensional, autonomous, first-order discrete dynamical system:

where $f=h\circ g$. In the simple case in which $f$ is a linear function, i.e. $f(\mu_{t})=k\mu_{t}+b$ for a linear coefficient $k$ and constant term $b$, Eq. 7 takes the form

For $k=0$, the user’s interest is not influenced by the recommender system but drifts with the constant $b$ – this case occurs in the real world with probability 0. For $k\neq 0$, the steady state equilibrium, obtained from solving the equation $\bar{\mu}=\bar{\mu}+k\bar{\mu}+b$, is given by $\bar{\mu}=-\frac{b}{k}$. Substituting $-\frac{b}{k}$ with $\bar{\mu}$ in Eq. 10 and taking the limit ${t\rightarrow\infty}$ on both sides, we obtain

Therefore, no matter how the system selects items, in the first three cases of Eq. (12) the user interest model over the item $a$ in question is always bounded over time. Of these, only the first case $|1+k|<1$, or equivalently $-2<k<0$, will occur with probability different from 0.

On the other hand, for $k>0$ the recommender system will degenerate strongly with $\mu_{t}$ growing at an exponential rate. For $k<-2$, the recommender system will degenerate weakly with $\sup_{i<t}\mu_{i}$ growing exponentially.

In summary, we can draw the following conclusions when an item $a$ is served i.o.: 1) for $k>0$ or $k<-2$, the user interest $\mu_{t}$ degenerates by growing exponentially, and therefore the recommender system needs to exert control on how frequent such items are shown to the user in order to control the speed of degeneracy of $\mu_{t}$; 2) in all other cases, the user interest $\mu_{t}$ does not degenerate ($-2\leq k<0$), or the system cannot control linear degeneracy ($k=0$, improbable case).

Non-linear Deterministic Model

If $f$ is a non-linear deterministic function, the steady state equilibrium is reached at zeros of $f$. Sufficient (but not necessary) conditions for global stability are given by the following theorem (Galor, 2007):

then a stationary equilibrium of the difference equation $y_{t+1}=g(y_{t})$ exists and is unique and globally (asymptotically) stable.

then it degenerates strongly as $t\rightarrow\infty$.

We prove that $\mu_{t}\rightarrow\pm\infty$ as $t\rightarrow\infty$. At time $t_{0}$, either $\mu_{t_{0}}>d_{t_{0}}$ or $\mu_{t_{0}}\leq d_{t_{0}}$. First we consider the case where $\mu_{t_{0}}>d_{t_{0}}$. At the next time step $t_{0}+1$, there are again two different cases:

$d_{t_{0}+1}<d_{t_{0}}$: it implies that $\mu_{t_{0}+1}>\mu_{t_{0}}>d_{t_{0}}>d_{t_{0}+1}$ by Condition 14.

$d_{t_{0}+1}\geq d_{t_{0}}$: by Condition 15, $d_{t_{0}+1}-d_{t_{0}}\leq\mu_{t_{0}+1}-\mu_{t_{0}}$ and thus

using $\mu_{t_{0}}-d_{t_{0}}>0$. Hence in both cases we have $\mu_{t_{0}+1}>d_{t_{0}+1}$. Applying the same argument to every time step, we also have

By Condition 16, we conclude that if $\mu_{t_{0}}>d_{t_{0}}$ then $\mu_{t}\rightarrow\infty$ as $t\rightarrow\infty$.

In the other case, $\mu_{t_{0}}\leq d_{t_{0}}\Longleftrightarrow\mu_{t_{0}+1}\leq\mu_{t_{0}}$. Similarly at time $t_{0}+1$, we have either $d_{t_{0}+1}<d_{t_{0}}$ or $d_{t_{0}+1}\geq d_{t_{0}}$. Following the same argument as above and reversing the inequality signs, we have

By Condition 16, we conclude that if $\mu_{t_{0}}\leq d_{t_{0}}$ then $\mu_{t}\rightarrow-\infty$ as $t\rightarrow\infty$. ∎

Scale-invariance

Let $\mathcal{M}$ be the candidate pool of items, which is also the domain of function $\mu_{t}$.

If we define $\nu_{t}=\psi\circ\mu_{t}$, then all sufficiency theorems readily apply to $\nu_{t}$ and the conclusions hold for $\mu_{t}$ since $\|\nu_{t}-\nu_{0}\|_{2}=\|\psi\circ\mu_{t}-\psi\circ\mu_{0}\|_{2}$.

Appendix C Degeneracy Speed Analysis

We analyze the system degeneracy $\|\mu_{t}-\mu_{0}\|_{2}$ (when the candidate pool $\mathcal{M}$ is finite) and $\sup_{a\in\mathcal{M}}|\mu_{t}(a)-\mu_{0}(a)|$ (when $\mathcal{M}$ is infinite) asymptotically in the order of $t$. As $t\rightarrow\infty$, $\mu_{t}\rightarrow\infty$ for the selected items. Thus the selected items are asymptotically almost surely clicked, and hence for any such item $a$, $|\mu_{t}(a)-\mu_{0}(a)|\approx\delta(a)\cdot T_{a}(t)$.

In the Random recommender system, $T_{a}(t)\approx tl/m$ and thus

Both the Oracle and the Optimal Oracle have a fixed set of items $\mathcal{S}$ that they keep selecting. Thus $T_{a}(t)\approx t$ for any item $a\in\mathcal{S}$ and

where $m^{*}$ are the number of close to optimal arms. Therefore all five models have the degeneracy quantity $\|\mu_{t}-\mu_{0}\|_{2}$ converge to a linear function of $t$.

Infinite Candidate Pool.

Since we sample $\delta(a)$ from a bounded interval $[-b,b]$, asymptotically $\sup_{a\in\mathcal{M}}\delta(a)\approx b$

where $a^{*}$ is an item with $\delta(a^{*})\approx b$.

In the Random recommender system, $T_{a}(t)\approx const$ and thus

The Oracle has a fixed set of items $\mathcal{S}$ that it keeps selecting. Thus $T_{a}(t)\approx t$ for any item $a\in\mathcal{S}$ and

The Optimal Oracle model will pick all items with $\delta(a)\approx b$ and $T_{a}(t)\approx ct$ for some constant $c$.

For UCB and Thompson sampling the situation is less clear. When growth of the candidate pool is linear, then UCB will spend most of its time exploring new items and consequently degeneration is very slow. Thompson sampling will continue to play degenerate items with reasonable probability and so degeneration speed will be larger than for UCB. Precisely quantifying the rates of degeneration depends in a complicated way on the details of the model.