Incentivizing High-Quality Content in Online Recommender Systems

In an emerging line of work, [4, 6, 24, 21] have proposed game-theoretic models for producer incentives for content creation in recommender systems. We build on the $D$-dimensional model of [24] where producers select the quality level (magnitude) of their content in addition to the genre (direction). Several recent works introduce learning into the framework in different ways. For example, [5, 36] incorporate that producers running learning algorithms to converge to an equilibrium. [13] incorporate that the platform runs retraining, but assumes that both the platform and producers are myopic; in contrast, our work captures that producers non-myopically react to the platform’s learning algorithm.

Our work closely relates to a line of work [15, 14, 28] that aims to incentivize the creation of high-quality content. These works assume each producer creates a single digital good with fixed quality across time; thus, the platform’s learning problem is cast within the stochastic bandit framework. In contrast, we allow producers to create new content at each time step and thus can vary their quality over time, which leads our problem to an adversarial bandit framework. Another difference is we allow for $D$-dimensional content vectors which enables us to capture producer specialization, whereas they focus only on $1$-dimensional content quality.

Other aspects studied in this research thread include participation decisions by creators [29, 7], incentivizing exploration of bandit algorithms [26], strategic user behavior in recommender systems [19], and preference shaping [8, 11].

Our model can be viewed as a Stackelberg game where the leader (platform) wants to learn the best responses of competing, non-myopic content followers (producers) from repeated interactions. There has been a rich literature on learning in Stackelberg games, including security games [3, 17], strategic classification [18, 12, 9, 40, 17], and contract theory [39]. However, most of these works focus on a single follower who (myopically) best-responds to the leader at each round or makes a gradient update, while our work allows for non-myopic followers who compete with each other (see Section 2.1 for additional discussion).

One notable exception is [17], who consider non-myopic agents. Interestingly, their approach – delaying the use of feedback from the followers in learning – fails to incentivize high-quality content in our setting. The fundamental difference is that [17] aim to disincentivize gaming whereas we aim to incentivize investment. Moreover, while [17] focus on a single follower, our work studies competition between followers. [10] also studies non-myopic agents, but focuses on the case of a (single) fully non-myopic follower.

We build on the rich literature on online learning algorithms [20]. Most relevant to our work is the adversarial linear contextual bandits framework [30], which extends the stochastic linear contextual bandit setup of [2] and [1] to adversarial losses. In comparison to classical adversarial contextual bandits [31, 33], this framework places a linear structure on the losses. Our work builds on this framework and employs the linear variants of EXP3 proposed by Neu and Olkhovskaya [30] that achieve $O(\sqrt{T})$ regret. We also consider a full-information version of this framework.

Our model incorporates key aspects of online recommender systems and the resulting creator economy. For example, our model of users and content as $D$-dimensional embeddings captures the multi-dimensionality of user preferences and content attributes. In fact, high-dimensional user and content embeddings are implicitly learned by many real world recommender systems, which use two-tower embedding models based on matrix factorization [23]) or deep learning variants [37, 35]. Moreover, producers choosing new content embeddings every time step captures that real-world creators frequently upload new content (e.g. to their content channel on YouTube³³3See https://www.youtube.com/creators/how-things-work/getting-started/.). The cost function $c(p)={\Vert p\Vert }^c$ captures that real-world production costs scale with effort and are also independent of consumption because goods are digital [15, 24]. Finally, the assumption on platform information – i.e., that the platform make recommendations before observing content in the current round – captures that the platform faces a cold-start problem for new content due to its reliance on user feedback to learn about content [34].

However, our model does make several simplifying assumptions for mathematical tractability. For example, in our model, producers select all content at the beginning of the game, which from the online learning perspective means that the producers are oblivious adversaries, who cannot adapt to the randomness of the algorithm. Additionally, our model assumes that a single user arrives at every round, but this assumption is made for ease of exposition: our results directly generalize to the case where all users arriving at every round are drawn from the population. Finally, our negative results (Section 3) focus on standard no-regret learning algorithms such as LinHedge and LinEXP3 (see Section 3.3 for additional discussion). Despite these simplifying assumptions, our model is rich enough to provide insight into how a platform’s learning process impacts creator incentives over time.

Our model is a (generalized) Stackelberg game, where the platform is the leader and each producer is a follower. Stackelberg games are a general paradigm with many applications beyond content recommender systems, ranging from security games [25] to power grids [38] to marketing [27]. Our generalized Stackelberg game has several unique features: the leader (platform) is learning over time, the followers (producers) are non-myopic and take into account future time steps when choosing their actions, and multiple followers compete with each other leading to a more complicated equilibrium analysis. Since our work is motivated by online recommender systems, our model places several additional structural assumptions on the game: the follower’s action space is $D$-dimensional, the leader’s action is to pick one of the followers, the leader’s utility function is linear in the followers’ actions, and increasing “effort” is costly to the follower but beneficial to the leader. An interesting direction for future work would be to extend our model and findings to more general Stackelberg games, relaxing the structural assumptions on the utility functions and action spaces.

We establish an upper bound on producer effort at Nash equilibrium if the platform runs LinHedge. The bound depends on the time step $t$, learning rate schedule ${\left\{{\eta }_t\right\}}_{t\in T}$, discount factor $\beta$, exploration parameter $\gamma$, and cost function exponent $c$.

To interpret the bound, we apply typical learning rate schedules for no-regret algorithms. First, for the standard decaying learning rate schedule ${\eta }_t=O(1/\sqrt{t})$, Theorem 1 implies $\Vert p_j^t\Vert =t^{-\mathrm{\Omega }(1)}$. This reveals that the equilibrium quality will be low in the long run and in fact approach zero as $t\to \mathrm{\infty }$. For the standard fixed learning rate schedule, ${\eta }_t=\eta =O(1/\sqrt{T})$,Theorem 1 implies $\Vert p_j^t\Vert =T^{-\mathrm{\Omega }(1)}$ for each $t\le T$. This similarly vanishes as $T\to \mathrm{\infty }$, and is actually even more pessimistic than the bound for the decaying learning rate schedule.

The intuition for why producers create low-quality content is that a small learning rate reduces the impact of a producer’s action on their future probability of being recommended. Thus, when the producer utility is discounted, the cost of putting in effort at a given time step outweighs the benefit of being recommended a bit more frequently in the future.

Our bound also illustrates how the discount factor $\beta$ and the cost function exponent $c$ impact content quality. First, the bound decreases in $\beta$, approaching zero as $\beta \to 0$. The intuition is that since content created today affects recommendations only in the future, more myopic producers have less incentive to exert effort at the current round. Second, the bound increases in $c$, approaching $0$ as $c\to 1$ and approaching $1$ as $c\to \mathrm{\infty }$. The intuition is a larger $c$ makes it cheaper to create content with norm .

We establish an analogue of Theorem 1 for LinEXP3, which similarly implies that typical learning rate schedules cause low-quality content at equilibrium. In particular, the quality decreases at a rate $t^{-\mathrm{\Omega }(1)}$ for ${\eta }_t=O(1/\sqrt{t})$, and at rate $T^{-\mathrm{\Omega }(1)}$ for ${\eta }_t=O(1/\sqrt{T})$.

The extra factor of $P$ in the bound of Theorem 2, compared to Theorem 1, arises from analyzing the additional randomness in bandits versus the full-information setting. The arm pulled at any time step influences the probability distribution of future arms being pulled, complicating producers’ choices. However, this extra $P$ may be an artifact of the analysis rather than a fundamental difference between the two settings.

The constraint in (4) guarantees that producers weakly prefer to stay in the active set over being punished at equilibrium. The slack $ϵ>0$ guarantees that this preference is strict. Altogether, we achieve user welfare $(1-ϵ)\cdot W(c,\beta ,𝒟,P)$, since the objective function in (4) is exactly the user welfare guaranteed by the individualized criteria. A formal proof is given in the full version [22]. $◀$

Theorem 7 shows that the welfare of PunishUserUtility as $ϵ\to 0$ is at least $W(c,\beta ,P,𝒟)$. To analyze the welfare of PunishUserUtility, it thus suffices to analyze $W(c,\beta ,P,𝒟)$.

In the following subsections, by analyzing the welfare $W(c,\beta ,P,𝒟)$, we show that PunishUserUtility addresses the two shortcomings of PunishLinDirectionHedge described in Section 4.3. More specifically, we show that the welfare $W(c,\beta ,P,𝒟)$ achieves an improved dependence on the number of producers $P$ (Section 5.2) and the user distribution $𝒟$ (Section 5.3).

We first analyze the welfare $W(c,\beta ,𝒟,P)$ as a function of $c$, focusing on the limiting case of $c\to \mathrm{\infty }$ for analytic simplicity.

Proposition 9 relates the limiting welfare of PunishUserUtility to the optimum of (5), which is easy to analyze. Moreover, the optimum of (5) is $1$ once the number of producers is as high as the number of users in the support (see the full version [22]). The intuition is that there are enough producers to specialize their content at the direction of each individual user so that all users are catered to. Finally, the optimum of (5) is always at least as large as $\Vert {𝔼}_{𝒟}[u]\Vert$ since we can set $p_j={𝔼}_{𝒟}[u]/\Vert {𝔼}_{𝒟}[u]\Vert$ for all $j$. We defer the full proof to the full version [22].

This analysis illustrates that while PunishLinDirectionHedge incurs a $\Vert {𝔼}_{𝒟}[u]\Vert$ factor from the lack of specialization, PunishUserUtility can achieve better dependence on the user distribution $𝒟$ when $P$ is sufficiently large. The economic intuition is that different producers can point at different directions of user subgroups rather than a single direction of the average user, allowing content to align with user preferences in a more fine-grained way.

Moving beyond PunishLinDirectionHedge, this analysis also enables us to compare the welfare of PunishUserUtility to arbitrary algorithms. The following corollary of Proposition 9 shows that the welfare bound $W(c,\beta ,𝒟,P)$ is optimal relative to any algorithm.

We defer the proof of Corollary 10 to the full version [22]

Since $W(c,\beta ,𝒟,P)$ lower bounds the welfare of PunishUserUtility in the limit as $ϵ\to 0$, Corollary 10 shows that PunishUserUtility in the limit as $c\to \mathrm{\infty }$ achieves the optimal welfare in this limiting regime. (We note that for any $ϵ>0$, PunishUserUtility is suboptimal due to the multiplicative factor of $(1-ϵ)$.) This result highlights that PunishUserUtility successfully leverages specialization to maximize user welfare.

Since the above analysis focused on the case of $c\to \mathrm{\infty }$, we now turn to finite $c$ and examine the gap between $W(c,\beta ,𝒟,P)$ and the upper bound in Proposition 12. For analytic tractability, we focus on the case of 2 users and explicitly compute $W(c,\beta ,𝒟,P)$ in closed-form.

We defer the proof to the full version [22].

Proposition 11 reveals that when (the two users’ preferences are similar), producers do not specialize their content to individual users even when $P$ is arbitrarily large. This contrasts with Proposition 9 where specialization occurred where the number of producers was as large as the number of users. This finding bears resemblance to [24] where specialization did not always occur at equilibrium when preferences were similar across users; however, the results in [24] focus on the equilibrium behavior in a static game, whereas we focus on the solutions to a welfare-maximizing optimization program with cost constraints.

To analyze the gap between $W(c,\beta ,𝒟,P)$ and the optimal welfare, we compare $W(c,\beta ,𝒟,P)$ with the following upper bound on the welfare of any algorithm (proof deferred to the full version [22]).

When comparing the bound in Proposition 11 with the bound in Proposition 12, there are two sources of suboptimality. First, even when $\theta \ge {\theta }^{\ast }(c)$, the lower bound is a factor of ${((1-\beta )/2)}^{1/c}$ smaller than the upper bound. Second, when there is an additional factor of ${\Vert 𝔼[u]\Vert }_2$ gap. This arises from the lack of specialization. Closing the gap between the upper and lower bound for finite $c$ is an interesting direction for future work.