Evaluation Perspectives of Recommender Systems: Driving Research and Education

Fairness becomes a critical factor when recommender systems are deployed in settings where harmful discrimination may occur. We distinguish between different classes regarding “who” fairness might concern [1, 18]. Consumer side fairness or user side fairness ensures that consumers¹⁰¹⁰10“Consumer” is commonly used to indicate the people using a recommender system. The term should not be used when the recommender system recommends people, such as in dating applications or job recommendations. For brevity and clarity, we will use consumer in this piece as we do not explicitly talk about these topics. of the recommender system are treated fairly in the quantitative and qualitative aspects of their experience. This involves ensuring equity of utility or usability, fair representation, avoiding stereotypes, etc. Fairness towards item side entities ensures a fair treatment of items; it can include provider and subject side fairness but can also be considered without knowledge of providers or subjects. A system can be unfair by treating similar items differently, e.g., when two news articles on the same topic and with comparable quality are not exposed equally. Provider-side fairness is an item-side entity concern which ensures fair treatment of item providers. Subject-side fairness is an item-side entity concern which ensures fair treatment of the subjects (people or entities) mentioned in, or related to the items. For example, in news recommendation, a system can be unfair to the gender of people described in news articles or to specific topics discussed in the articles. Multisided fairness [11] considers consumers and providers, demanding fairness on both sides.

Fairness is often characterized as individual vs. group fairness [17]. The goal of individual fairness is to treat similar individuals similarly, so that each individual receives an appropriate treatment in accordance with some task-specific notion of “merit”. The goal of group fairness is to treat different groups similarly, so that there are no systematic disparities across groups. Usually, a protected group is contrasted against an unprotected group (also called dominant or majority group) to guarantee that protected individuals are treated comparably to unprotected ones. Groups are often defined upon attributes from anti-discrimination law, e.g., gender, ethnicity, religion, and age.

Individual fairness assumes a function to measure the similarity among individuals. Defining such similarity function is challenging due to the lack of ground truth, data biases, task dependency [25] and very often results in solving the task itself [12]. For example, a “perfect” similarity function based on user preferences and past interactions could be used to generate “perfect” recommendations. While group fairness might seem easier to accomplish, it requires access to protected attributes to define groups. These attributes are often unavailable or difficult to collect because they represent sensitive data, e.g., gender. Moreover, group fairness does not guarantee fair treatment among individuals within a group due to aggregation and comparison among groups (fairness gerrymandering [32]). For example, a music recommender system might achieve group fairness with respect to gender by increasing exposure for a single artist, but this does not ensure fairness for other artists of the same gender.

Exploring the “How?” of fairness involves examining various dimensions through which fairness can be achieved or compromised. Here, we refer to some examples of how unfairness can lead to unfair distribution of utility, severe consequences, exposure, discrimination, misrepresentation, and reinforces stereotyping.

Unfair distribution of utility Unfairness in recommender systems can lead to unequal distribution of utility, where benefits such as opportunities are disproportionately allocated. When recommendations favor certain consumers/users or item providers over others due to biases in data or algorithms, some groups receive more exposure and advantages, while others are marginalized [22, 19, 24]. This inequitable distribution not only reduces the overall satisfaction and utility for disadvantaged users but also perpetuates existing inequalities and limits diversity.

• $\mathrm{■}$

  How can recommender systems be designed to ensure an equitable distribution of utility among all users/items/subjects?

• $\mathrm{■}$

  What factors contribute to the unfair distribution of utility in recommender systems?

• $\mathrm{■}$

  How do biases in the data and algorithms affect the distribution of utility among different user/item groups?

• $\mathrm{■}$

  What metrics can be used to measure the fairness of utility distribution in recommender systems?

• $\mathrm{■}$

  How can interventions be implemented to correct the unfair distribution of utility in existing recommender system algorithms?

Disparity of Exposure Depending on the user attention model that is considered, an item’s position in the recommendation list determines the exposure of individuals or groups of items [7, 43]. Therefore, exposure has effects and implications on how much users will consume those individual or groups of items. Disparity of exposure is typically based on the principles of equality of opportunity. This can be further operationalized in different ways [15, 31].

For example, disparity of opportunity can be based on the idea that all item groups/similar items should get exposure proportional to the collective merit of the items in the group or the merit of individual items [30]. Fairness for individuals can be defined following the idea that exposure should be proportional to relevance for each subject in a system. In contrast, fairness for groups means that exposure should be equally distributed among members of groups defined by sensitive attributes such as gender and lyric language [43].

• $\mathrm{■}$

  How can exposure be measured and balanced to ensure fairness for all users and item providers?

• $\mathrm{■}$

  What algorithms or techniques can be used to ensure equitable exposure?

• $\mathrm{■}$

  How does unequal exposure affect user satisfaction and engagement with recommender systems?

• $\mathrm{■}$

  What are the challenges in achieving group-level exposure fairness, and how can they be addressed?

• $\mathrm{■}$

  How can exposure fairness be maintained over time as user preferences and content availability change?

Discrimination occurs when the algorithmic decisions tend to disadvantage certain groups based on characteristics such as demographic information, e.g., ethnicity, gender, age, or socioeconomic status [2].

• $\mathrm{■}$

  How does discrimination affect user trust and platform credibility?

• $\mathrm{■}$

  What are the legal and ethical implications of discrimination in recommender systems?

• $\mathrm{■}$

  How can inclusive data collection practices reduce the risk of discrimination in recommendations?

Misrepresentation refers to an inaccurate representation of users or item providers’ characteristics [21, 17]. Misrepresentation can be in the form of inaccurately representing users’ interests and information needs internally, preventing certain user groups from systematically having less accurate representations (e.g., user embeddings or other user models that may lead to stereotyped recommendations [21]. Providers can be harmed by not having their products consumed.

• $\mathrm{■}$

  How can misrepresentation in user profiles and item descriptions be identified and corrected in recommender systems?

• $\mathrm{■}$

  What impact does misrepresentation have on user satisfaction and item provider success?

• $\mathrm{■}$

  How do inaccurate user models contribute to the spread of stereotypes in recommendations?

• $\mathrm{■}$

  What techniques can improve the accuracy of user and item representations to prevent misrepresentation?

• $\mathrm{■}$

  How can transparency in recommender systems help users understand and correct potential misrepresentations?

Reinforcing stereotype refers to the potential of recommender system algorithms to perpetuate harmful or unnecessary stereotypes by consistently promoting content that aligns with narrow, stereotypical views [38].

• $\mathrm{■}$

  How do recommender systems contribute to the reinforcement of societal stereotypes?

• $\mathrm{■}$

  What are the long-term impacts of stereotype reinforcement on users and society?

• $\mathrm{■}$

  How can algorithms be designed to avoid reinforcing stereotypes?

• $\mathrm{■}$

  What role does diverse and inclusive data play in preventing stereotype reinforcement? How can user feedback be used to identify and mitigate the reinforcement of stereotypes in recommendations?

Machine learning models often optimize some static objectives, causing fairness to be regarded as a static function. Most definitions consider fairness as a one-shot process, i.e., with respect to a single point in time. The underlying assumption is that fairness will be beneficial for the protected individuals or groups, as well as the whole society, in the long term. However, decisions based on ML models can be iterated over time, and one-step fairness can even cause harm [28, 34, 35, 13, 33, 6, 24].

Recommender systems are dynamic and interactive by nature, i.e., the nature of entities may change over time. For example, groups based on attributes such as popularity can quickly change over time, and fairness interventions can potentially drive items into or out of the top popular group. This distinction of fairness as a long-term or short-term process results in static vs. dynamic fairness. Static fairness disregards changes in the underlying environment, e.g., utility, attributes, etc., while dynamic fairness adapts to the environment [26].

The severity of consequences refers to the negative impact of unfair recommendations on all entities involved, e.g., consumers, item providers, etc. For instance, severe consequences for consumers can be in the form of missed opportunities, financial losses, or psychological harm. Item providers such as content creators or sellers can face severe consequences that manifest as reduced visibility and revenue (see Section 4.2.2 for concrete examples).

The extent to which unfair recommender systems can cause harm depends also on the temporal dimension. For example, disparity of exposure might not cause immediate harm but, if reiterated in the long-term, can potentially lead to severe discrimination, job and profit loss, and reinforcement of stereotypes. In the long term, unfairness can also have significant societal consequences. With news and social media sites, unfair recommender systems might promote content emphasizing only one political side or misinformation discriminating against certain groups [50, 5].

Academic search and recommendation aim to help researchers find relevant papers for their interests. The widespread use of these systems calls for ways to ensure fair information access to avoid harmful consequences to authors, institutions, and journals. In Fig. 2, we briefly overview the main concepts behind fairness for the use case “research paper recommender systems”.

Possible actors involved are paper authors, users of the search or recommender system, research institutions, and publishing venues, e.g., conferences and journals. Author group fairness can be defined by attributes such as gender, seniority, geographical origin, or discipline. The Gross Domestic Product (GDP) and the country can apply to research institutions and country for publishers.

Examples of fairness concerns for this domain include:

• $\mathrm{■}$

  If the system provides an unfair disadvantage to a group of authors, this may lead to lower recognition in the field for this group of authors (discrimination, disparity of exposure, misrepresentation). Consequently, this can lead to challenges for them in finding a job posting in academia and a loss of revenue in the long term.

• $\mathrm{■}$

  If a discipline is under-represented, this can lead to a knowledge gap for the user (reader) of the system (disparity of exposure, misrepresentation). This knowledge gap can lead to less-informed papers and potential rejection of the work.

• $\mathrm{■}$

  If there is a systemic bias on the location or renown of an institution, this can lead to under-recognition for these institutions (discrimination, disparity of exposure, misrepresentation), thus stumping their growth, and harming their search for funding and students.

• $\mathrm{■}$

  If articles from a publisher or group of publishers are under-recommended (discrimination, disparity of exposure, misrepresentation), this can lead to a lower value for publications by this publisher and consequently to fewer submissions to the journal, leading to diminishing value for the publisher.

Online retailers provide users with easy access to products from all over the world. Online marketplaces such as Amazon, Zalando, and Ali-Express serve many users with products from various vendors. Thus, their recommender systems have an impact on the fairness towards many stakeholders. In Fig. 3, we briefly overview the main concepts behind fairness for the use case “e-commerce recommender systems”.

We identify two main classes of actors from the selling and buying side: companies involved in the production chain (manufacturer, vendor, shipping companies) and consumers. Meaningful attributes for companies are size and country. For consumers, we can consider gender, ethnicity, age group, and income level as relevant attributes.

Some specific concerns we would like to highlight are the following:

• $\mathrm{■}$

  If the system is under-recommending items from a group of vendors (discrimination, disparity of exposure, misrepresentation), this could lead to lower sales for these vendors. This, in turn, is likely to lead to a loss in revenue for them.

• $\mathrm{■}$

  If there is an unfair distribution of the manufacturing plants of recommended items, then underrepresented manufacturing plants might lose revenue as the items they make are not being sold as easily (discrimination, disparity of exposure, misrepresentation). This could lead to job loss for the employees and even bankruptcy.

• $\mathrm{■}$

  If one user group is consequently recommended more expensive items (discrimination, misrepresentation), this may lead to higher strains on their income; thus, introducing or reinforcing a monetary gap with the other groups.

• $\mathrm{■}$

  If recommendation quality is systemically lower for a group of users (unfair distribution of utility, misrepresentation), this leads to lower utility for them.

• $\mathrm{■}$

  If the recommender system consistently recommends stereotypical items to groups of users, this can lead to reinforcing stereotypes. For example, girls might get recommended books about princesses, while boys get books about knights.

Operationalization must begin with a clear scope of what is to be evaluated. This typically needs to be the end-to-end system; because fairness does not necessarily compose [16], we cannot assume that improving the fairness in some respect for one component of the system will necessarily improve fairness of the system’s final output or impact. While it is vital to study different stages and components (e.g., candidate selection [10] or embeddings [47]), they cannot be studied only on their own; downstream impacts are crucial to understanding their contributions to fairness in the system’s social impact.

The scope of measurement, therefore, consists of several aspects (some of which are decided in earlier stages, such as problem definition; see Section 4.2.3):

• $\mathrm{■}$

  What component(s) or intervention(s) are being evaluated? Some projects will be purely descriptive, seeking to understand the fairness of a current system; others will be incorporated into evaluations of changes proposed for other purposes (e.g., ensuring a model intended to improve user modeling accuracy does not induce unfairness); and still others are to evaluate the effectiveness of a fairness intervention. The scope of measurement needs to be in line with the problem definition (Section 4.2.3) and specified in more (fine-grained) detail.

• $\mathrm{■}$

  What system aspect(s) are to be evaluated? As noted above, this usually needs to include fairness of the final system outputs or impacts, but it may also include targeted measurements of other components. For example, an experiment on improving the fairness of candidate selection in a multi-stage research paper recommender system should measure both the fairness of the selected candidates, and the fairness of the final rankings, to assess both (1) if the intervention is behaving as it is intended to (akin to a manipulation check in other research designs) and (2) if it is having the desired effects on the surrounding system.

• $\mathrm{■}$

  What entity classes are to be considered? This flows from the selection of stakeholders (see Section 4.4), but operationalization needs to produce a specific metric for users, items, providers, or other entities in the data model; and further, the evaluator must decide whether it is being computed over all entities of that class or a subset of the data. The unit of analysis [44] and aggregation strategy are also important.

At a high level, there are two major computational and data inputs to an evaluation: the system to be evaluated and the data to be used for that evaluation. The system is common to all evaluation types, as is some of the data (consumption or feedback data, content, etc.).

Fairness evaluations often require additional data, particularly for group fairness, where group membership data is required. There is a variety of sources for such data:

• $\mathrm{■}$

  Integrate additional public data sets. For example, [19] combine three external data sources with book consumption data to measure author gender fairness for book recommendations.

• $\mathrm{■}$

  Obtain data from additional sources, such as data markets. Depending on the data source, this may bring significant privacy, ethics, and legal questions.

• $\mathrm{■}$

  Collect or produce data, e.g., by paying for expert data annotations and metadata preparation.

• $\mathrm{■}$

  Use background data available in the specific domain or related domains. Background data, such as demographic information, social indicators, or historical trends, can be a valuable source to fill gaps and enrich the context. Proper validation and alignment with the primary data source are crucial to ensuring that the background data contributes meaningfully.

Great care is needed to appropriately annotate data, particularly for ascribing potentially sensitive identity characteristics to people. For example, the US Program for Cooperative Cataloging has developed recommendations for discerning and recording authors’ gender identities [8]. These recommendations disallow inference of gender identity from names or photos, in favor of authors’ explicitly-stated identity (preferred) or inferences from pronouns in official biographical material they approved (if the author describes themselves with the pronoun “her”, for example, the guidelines allow that as evidence of a female gender identity). Automated inference, while appealing computationally, has significant challenges in terms of its accuracy and fairness as well as ethical and conceptual concerns about its reification of specific ideas of gender and its (dis)respect for autonomy and right to self-identification among the people identified [27, 37]. Each identity has a different set of considerations (which may vary between cultures and regions, for example, in the different ways racial categories function in different countries). However, a similar concern is required for any categorization of people. There are also a range of privacy and regulatory concerns, in some cases prohibiting data collection and in others requiring it [3].

Once the data has been sourced, either internally or externally, operationalization further depends on the nature and encoding of the data. Several key questions about group membership or other fairness-related data attributes affect further design choices, including:

• $\mathrm{■}$

  How complete is the data?

• $\mathrm{■}$

  What biases are in the data? This can be biases in values, biases in errors (e.g., job candidates of particular races are more likely to have erroneous labels), and biases in selection (e.g., label-dependent selection bias [14], where certain label values are more likely to be observed).

• $\mathrm{■}$

  How many and what categories are in the data? E.g., does it only have binary gender, or does it represent non-binary gender identities as well [37]?

• $\mathrm{■}$

  How are entity categories represented? Are they discrete, or does the data represent mixed, partial, or unknown membership?

The overall design of the experiment – data splitting, running systems, etc. – for fairness evaluations is not significantly different from other evaluations for accuracy, diversity, novelty, etc., except for the need to incorporate additional data for some fairness constructs. The guidance elsewhere in this report, therefore, applies.

The actual specific measurements or objectives used to quantify fairness need to align clearly with the problem, the nature of the constructs involved in the problem (e.g., effectiveness or gender), and the practicalities of the data used to compute them.

For example, several metrics for both provider- and consumer-side fairness only operate on discrete binary attributes in which membership is fully known and are therefore difficult or impossible to apply to more realistic settings with multiple groups and unknown or partial membership [39]. This is misaligned with the nature of the construct (many characteristics are not binary), as well as the data practicalities (complete data is extremely rare). Metrics for individual item fairness suffer from other limitations, e.g., they cannot be used to assess systems in isolation but only for relative comparisons across systems [40, 41]

Some of the things that need to be considered for measurement selection include:

• $\mathrm{■}$

  The metric should be a plausible approximation of the problem. This is the most critical consideration because a metric that is computable but does likely not map to the problem likely is not measuring the intended issue.

• $\mathrm{■}$

  For group fairness, the number of groups and the nature of membership [39]. This affects several things, including whether differences or ratios are appropriate, or whether a different way to compare values is needed [23].

• $\mathrm{■}$

  The nature of the impact or resource to be fairly allocated, such as whether it is subtractible (allocation to one person comes at the expense of another) [17, 20]. Zero-sum operationalizations of non-subtractible goods, such as consumer-side utility (one users’ good recommendations do not affect another users’ bad ones), induce competition where it need not exist [21, 20]. [45] address this for consumer-side equity of utility by using an positive-sum metric, the sum of the logs of the total utility for each group, that has optimal reward gain from improving utility for the least-well-served group.

• $\mathrm{■}$

  Metrics should deal in a clear and documented manner with missing data (feedback, group annotations, or other data).

• $\mathrm{■}$

  Metrics and their aggregations should respond well to edge cases such as empty lists, empty groups, etc.

• $\mathrm{■}$

  Whether or not there is a specific target, and if so, what that target is, needs to be clearly specified.

• $\mathrm{■}$

  How fairness should relate to other concerns, such as utility, when appropriate. For example, pursuing equal exposure for items, providers, or groups and exposure proportional to (estimated) utility will yield different metrics [39, 7].

Further, metrics differ in their interpretability and scope of comparability: some can measure fairness in a way that is comparable across data sets or target distributions. The Gini coefficient, for example, is a data-independent measure of resource concentration, and can be used to document that exposure is more heavily concentrated on a smaller set of items in one system or data set than another. On the other hand, expected exposure loss [15] cannot be directly interpreted and can only assess which of several systems better matches the target distribution.

In some cases, it is not necessary to directly measure unfairness, depending on the evaluation goals. Disaggregated evaluation [4, 22] – grouping entities by attribute and computing metric separately for each group – is useful in its own right to assess whether one group is getting greater benefit or harm than another, even without quantifying the difference itself. Distributional evaluation [27] takes this further, looking at distributions across individual entities or within entity groups (e.g., looking at the distribution of utility for consumers of different genders).

Fairness evaluation is not a linear process that can proceed from definition to operationalization to further stages without detours or backtracking, but is often an iterative process. The operationalization needs to be checked against the problem definition to ensure that it accurately captures the construct of interest.

Also, this check should not be done solely by the research team. Following the idea of member checking in qualitative research [9], it is helpful to return to the stakeholders involved in the problem definition to engage them in assessing whether the proposed design captures the concerns they articulated.

Reproducibility refers to the ability to achieve the same findings as the original researchers utilizing existing data from a prior study [20]. In other words, reproducibility describes the minimum necessary information to re-implement, re-execute, repeat, and replicate experiments to verify the findings described in a scientific study [11, 20, 36].

While different and partially incompatible definitions of reproducibility and related concepts exist, in computer science, reproducibility often implies that experiments, including data processing steps, can be accurately repeated to produce the same results. This typically involves making code and data publicly accessible and providing detailed documentation of computational methods and algorithms. This section adopts this notion as a working definition of reproducibility.

Generally, reproducibility in recommender systems research is essential, as it allows researchers to ensure that others can (a) verify previously published results and (b) make sure that their algorithmic contributions truly help to advance state of the art.

To shed light on our concerns, we list several issues often observed in papers concerning reproducibility. First and foremost, while we observed a positive development over the past decade, many researchers and practitioners still do not publish their algorithmic implementation. While Intellectual Property (IP) rights issues may pose a challenge in some cases, it is still important that the implementations are made available to others.

However, publishing the proposed model or method implementation is not enough. For reproducibility, the code for the entire evaluation pipeline is required, starting with loading the data and ending with results. When doing so, one should focus on the proposed model and the implementation and training of the used baselines. Specifically, tuning the baselines, i.e., how the hyperparameters were determined, should be made public.

Scientific evaluation is a systematic approach used to assess the validity of a hypothesis. Evaluation methodology outlines the proper conduct of scientific evaluation [14]. The evaluation methodology is typically characterized by a concrete set of steps to ensure rigorous scientific standards set by the community. It is driven by the underlying hypothesis or research questions to ensure that assessments and conclusions drawn from the empirical results are scientifically sound [34]. For a given research problem, researchers commonly make decisions in their evaluation methodology that are inspired and justified by previous research.

The evaluation methodology in offline experiments for recommender system research typically details the collection and preprocessing of data, the chosen baselines, the learning and optimization strategy, the metrics used to compare methods, and the method used to analyze the results.

The evaluation methodology is critical in scientific work because it allows researchers to create an evaluation process that others can validate and reuse for similar research. An extensive description of the evaluation methodology is also essential for the peer-reviewing process. It allows reviewers and other researchers to critically assess the validity of the results obtained to confirm or refute a hypothesis. The peer-reviewing process should thus ensure that the evaluation methodology in a research paper is rigorous and follows the community’s consensus.

In the following, we identify a few common problematic recommender systems evaluation methodology practices that our guidelines intend to remedy. First, we find that researchers often provide little or no justifications for certain choices of their research methodology. Researchers often justify certain decisions by arguing that the decision is common practice or that another group of researchers did it. Adopting methodological choices from previous works can be beneficial, making research more comparable. However, such a justification may often not be scientific or complete, e.g., when the the common practice had no proper justification either.

Second, we note that data leakage is a common problem in evaluation methods that may be unseen by just reading the evaluation methodology. Data leakage refers to using the test split or knowledge about the test split in the training process [21].

Finally, while hyperparameter optimization is a standard procedure in recommender systems research, the configurations are often not shared or are incomplete. However, it has been shown [38] that the performance difference between configurations can be significant and change the ranking of algorithms. This is especially problematic in research that claims to improve over the state-of-the-art but provides incomplete hyperparameter tuning specifications for baseline algorithms.

We organize the proposed questions on reproducibility (and their corresponding explanations) in the following groups.

1. 1.

   Code-related Aspects: Is the code of the full experimental pipeline publicly available?
   Sharing all artifacts needed or used to obtain the numerical results reported in a paper is essential for reproducibility. Appropriate documentation must also be provided so other researchers can re-execute the experiments. If possible, an execution-ready environment, e.g., in terms of a Docker container, should be made available.

   1. [1.1.]

   2. (a)

      Code of proposed algorithm/framework/method/model

   3. (b)

      Code of all baselines

   4. (c)

      Code for preprocessing and postprocessing

   5. (d)

      Code for hyperparameter tuning

   6. (e)

      Code for execution (training and testing)

   7. (f)

      Code for statistical analysis

   8. (g)

      Documentation and installation/execution instructions

2. 2.

   Data-related Aspects: Is all relevant data publicly available?
   Reproducibility is only possible if the data used as input to the models and the results are publicly available. It may be insufficient to provide pointers only to previously published datasets, e.g., because preprocessing steps have been applied or publicly shared datasets are sometimes updated.

   1. [2.1.]

   2. (a)

      Original datasets

   3. (b)

      Preprocessed datasets

   4. (c)

      Train/validation/test splits

   5. (d)

      Results (outcomes of measurements)

   6. (e)

      Trained models

3. 3.

   Configuration Aspects: Are all relevant configuration parameters reported?
   Besides code and data, the specifics of the execution of the experiment must be documented. This concerns how the models were tuned, the execution environment, and its configuration.

   1. [3.1.]

   2. (a)

      Hyperparameter search strategy, search space and search time for all models

   3. (b)

      Optimal hyperparameters per dataset and model

   4. (c)

      Train-test splitting configurations

   5. (d)

      Random seeds

   6. (e)

      Required external libraries and their versions

   7. (f)

      Used hardware (configuration)

4. 4.

   Experiment specific aspects and other questions
   Depending on the specifics of the experiment, information about various other aspects should be provided. These questions should help better to gauge the level of reproducibility of the experiment. Further, these questions may serve as a place for researchers to justify certain technical choices.

   1. [4.1.]

   2. (a)

      Has an existing evaluation framework been used? If not, why not?

   3. (b)

      Is “one-click” reproducibility supported?

   4. (c)

      Are any instructions provided to reproduce (at least parts of) the experiment with limited hardware resources?

   5. (d)

      Is an expected runtime to reproduce the results provided?

For a mature research community, embracing the evaluation procedure used in previous papers can be considered good practice. However, we must acknowledge that several unjustified protocols, e.g., leave-one-out, have taken root in the recommender system community. Hence, justifying a research protocol only by saying it was used in previous papers is perhaps unreasonable.

1. 1.

   Research Questions and Hypothesis: Are the research questions and hypotheses expressed clearly and matching the method and the results?
   The research question should guide the development of the evaluation process. Therefore, it should be clearly stated, and the authors’ choices throughout the method should correspond with the research question and conclusions.

   1. [1.1.]

   2. (a)

      The research question is clearly stated.

   3. (b)

      The hypothesis is derived from the research question.

   4. (c)

      The experimental design is suited to address the stated research questions.

   5. (d)

      The conclusion is based on the research question and the experimental design.

2. 2.

   Baselines: Are baselines selected and tuned to ensure appropriate comparisons?
   While one should always compare to the latest best method for the particular task, it is also important to compare against earlier and probably simpler baselines to show the advantage of using the new, more complicated method.

   1. [2.1.]

   2. (a)

      The chosen baselines are appropriate to the hypothesis and research question.

   3. (b)

      One of the baselines is successful, e.g., state-of-the-art, for the given task.

   4. (c)

      At least one simple baseline, e.g., $k$NN, popularity, or random, is included.

   5. (d)

      The baselines are tuned. One must invest sufficient effort in properly training the baselines.

   6. (e)

      There needs to be clarity about whether the baselines were rerun or whether the results were taken from a previous paper.

3. 3.

   Evaluation Metrics: Is the chosen evaluation metric appropriate to answer the research question?
   Choosing the appropriate evaluation metric for the task is critical. Reporting a large number of unrelated metrics is not good practice.

   1. [3.1.]

   2. (a)

      The selected metrics are derived from the hypothesis, e.g., RMSE for rating prediction or precision$\mathrm{@}N$ for top-$N$ recommendation.

   3. (b)

      The reported metrics are not redundant, e.g., RMSE and MAE or DCG and NDCG.

   4. (c)

      Tradeoffs between the metrics are explained and evaluated.

4. 4.

   Data collection: Is the data collection process reasonable and well explained?
   This is appropriate when a new dataset is presented. This dataset may be collected from an already running system or using a particular user study.

   1. [4.1.]

   2. (a)

      The data collection process is clear.

   3. (b)

      The study participants’ recruitment, introduction, and participation incentives are explained.

   4. (c)

      Biases that exist in the data or arise from the data collection process are explained.

   5. (d)

      The used datasets are publicly available.

5. 5.

   Datasets: Are the chosen datasets appropriate for the task?
   In offline evaluation, choosing appropriate datasets is highly important. Using a diverse set of datasets supports claims for generalization. In cases where a particular domain is targeted, the datasets must be focused on the task.

   1. [5.1.]

   2. (a)

      The chosen datasets are appropriate to the task at hand.

   3. (b)

      It should be clear whether the datasets were chosen to demonstrate generalization.

   4. (c)

      In the generalization case, it is desirable to experiment with a sufficient number of datasets.

   5. (d)

      If showing the general applicability of a model is the goal, a diverse set of datasets is used.

   6. (e)

      The origins of public datasets are specified.

6. 6.

   Data preprocessing: Is the data preprocessing well justified and explained?
   It is often the case that researchers preprocess, prune, and filter the original dataset before training and testing. In general, preprocessing should be discouraged, especially the dataset’s filtering and pruning. Such preprocessing should be kept to a minimum and should be well explained.

   1. [6.1.]

   2. (a)

      If users or items were pruned from the dataset, the pruning is well justified.

   3. (b)

      When pruning is done because the evaluated method works better on a subset of the data, this is made clear.

   4. (c)

      : This process is clearly explained and justified if the data was converted, e.g., from numeric ratings to binary like/dislike.

7. 7.

   Data-splitting: Does the train-test split fit the structure of the dataset and the task? Most machine learning methods require a training phase. It is, hence, standard practice to split the data into training and test sets, where the test set is used only once to evaluate the algorithm once the training phase is done. The train-test split is designed to simulate the behavior at run time, when the system is aware of all information to date and must make future recommendations. Hence, the split procedure should correspond to the task at hand.

   1. [7.1.]

   2. (a)

      Typically, user-item interactions are split on time, where the training data contains the earlier interactions, and the test data contains the newer ones. When other types of splits are used, this is justified.

   3. (b)

      All algorithms are run on the same train-test split.

   4. (c)

      Cross-validation is applied when possible.

8. 8.

   Hyperparameter Optimization (HPO): Is the hyperparameter optimization procedure justified and appropriate for the task?
   For many machine learning methods, it is well known that HPO is a critical factor for performance. ML algorithms may underperform significantly when their parameters are not tuned to the dataset. Using an organized HPO process for all evaluated algorithms is highly important.

   1. [8.1.]

   2. (a)

      The optimization strategy is clearly stated.

   3. (b)

      The hyperparameter configuration space (parameter range) is sufficiently large and clearly defined.

   4. (c)

      The optimization time or number of tested configurations is clearly stated.

   5. (d)

      It is stated in case some algorithms are optimized differently.

9. 9.

   Experiment execution: Was the experiment executed such that the comparison results are fair and statistically sound?
   When running the experiments, all algorithms should receive equal treatment. Statistical significance should be computed to test the likelihood that the observed differences between the algorithms are real.

   1. [9.1.]

   2. (a)

      The boundaries between train and test data are respected (i.e., test data not used for checking convergence).

   3. (b)

      There is equal treatment of all compared algorithms (with respect, e.g., to HPO, runtime, hardware).

   4. (c)

      The statistical significance testing method is appropriate for the task.

   5. (d)

      The $p$-values are properly computed and reported.

   6. (e)

      Confidence intervals are provided whenever possible.

   7. (f)

      The hardware used in the experiment (e.g., memory, processor speed, GPU) is properly described.

10. 10.

    Sensitivity analysis: Did the authors conduct and report a sensitivity analysis concerning the method parameters and the dataset properties?
    Many algorithms have some parameters that must be tuned. It is important to analyze how different values for these parameters influence the performance. In many cases, an algorithm may also be sensitive to the dataset’s properties (e.g., sparsity).

    1. [10.1.]

    2. (a)

       The method is executed with different parameter values.

    3. (b)

       The values of all parameters are fixed except for the tested one.

    4. (c)

       The effect of the parameters on the method is reported and discussed.

    5. (d)

       If there are trade-offs between the parameters, they are made clear.

    6. (e)

       Sensitivity to dataset parameters is done similarly to the method parameters.

In this section, we provide several concrete suggestions that could be immediately implemented by the ACM RecSys program chairs, the ACM TORS¹⁵¹⁵15https://dl.acm.org/journal/tors/ editorial board, and the chairs and editors of related publications venues.

When addressing values in the context of multistakeholder recommender systems, economic and business-related values are often considered, especially for providers and system operators.

[9] provide a systematic review of value-aware recommender systems, introducing value primarily as an economic concept leading to monetary reward (i.e., profit and revenue). They distinguish several aspects that inform the value of monetary reward reflective of a business and economic view, including use value (e.g., increasing revenue by providing useful recommendations), estimated value (related to attractiveness and desirability, such as having a comprehensive music catalog to create recommendations from), cost value (e.g., the economic resources required to distribute a music album to the music streaming platform), and exchange value (the change in value over time, e.g., increase in a music artist’s recognition and popularity on the platform due to effective recommendations).

From this, we observe values related to user perception and customer loyalty, which are crucial from both a business and economic perspective. These values often relate to “the concepts of quality and personalization, experience and trust, features, and benefits” [9]. For example, in the music industry, a platform that provides highly personalized playlists based on users’ listening history can significantly enhance user satisfaction. This personalization not only helps users discover new music that aligns with their preferences but also fosters a sense of trust and loyalty towards the platform. Users are more likely to stay subscribed and recommend the service to others if they consistently experience high-quality, relevant recommendations.

In their work, [10] highlight that recommender systems typically serve an organization’s economic values. Besides profit and revenue (i.e., monetary rewards), this might be related to growth and market development. For example, music streaming platforms aim to generate profit and attract new users by offering social features like joint playlist creation, which benefit users when their peers are also on the platform. Furthermore, the authors characterize economic recommender systems as systems that exploit “price and profit information and related concepts from marketing and economics to directly optimize an organization’s profitability.” [35] identify strategic perspectives for both consumers and providers. For consumers, personal utility includes happiness, satisfaction, knowledge, and entertainment. For providers, organizational utility encompasses profit, revenue and growth. In addition, other values, such as changing user behavior to create demand might be relevant. For example, a music streaming platform might recommend emerging artists or newly released tracks to users, encouraging them to explore and adopt new music preferences, thereby creating demand for content that the platform can better monetize.

[41] examine the theory of business models in e-commerce recommender systems and identify the following value-driving aspects: efficiency (e.g., the exposure of music artists in recommendation lists or the number of clicks on recommended music tracks), complementarities (e.g., creating value through synergies by combining different item types like recommending merchandise articles along with track recommendations of a specific music artist), lock-in and churn prevention (e.g., retaining subscribed users by providing meaningful recommendations), and novelty and product planning (e.g., finding new fans through recommendations to users who might like an artist’s music or getting inspired to create new music album).

Beyond these economic and business values, societal and human-centric values, which cover other important aspects, are also crucial for businesses and platforms. These values will be discussed in the following section.

Societal and human-centric values for stakeholders in recommender systems focus on ensuring that these systems operate in ways that prioritize humans individually and society as a whole. We find that there are four themes of societal and human-centric values for stakeholders in recommender systems that are relevant in the light of evaluation: (i) usefulness, (ii) well-being, (iii) legal and human rights, and (iv) public discourse and safety [54, 55].

Usefulness and enjoyment means that recommendations should meet the needs and expectations of its stakeholders effectively and efficiently [28]. For example, in the case of a music recommender system, users should be able, via the recommender system, to discover new music that they might enjoy and match their taste. At the same time, usefulness refers to the recommender system’s ability to support music artists to get their outputs recommended to potentially interested listeners. Control and privacy is a closely related value that pertains to the degree of influence and customization stakeholders might have over the recommendations that are generated. This includes privacy aspects in a way that users might want to control their preference data that is shared with the recommender system [54].

Well-being refers to the recommender system’s ability to help its stakeholders to feel satisfied. In the case of a music recommender system, this means that recommendations should influence the experience with the music streaming platform positively, e.g., provide music recommendations to help listeners relax or relieve stress [27]. In this respect, well-being is related to emotional, mental, and physical health. Other related values are connection, community and social bonding, e.g., to enable users to connect with like-minded people or to enable music artists to contribute their outputs to a specific community. Thus, also reputation, recognition and acknowledgment might be valuable for some stakeholders, e.g., to support music artists in getting their contributions being recognized by music listeners [37]. Personal growth and development might also be values contributing to well-being in the sense that, e.g., music recommendations could help people explore new music styles and genres, supporting exploration and self-discovery [6].

Concerning legal and human rights, fairness may be an important value for stakeholders of a recommender system at evaluation time. For example, the music stream platform should aim to provide meaningful recommendations to all user groups, independent of, e.g., their musical taste or other demographic characteristics [22, 12]. Additionally, the music recommender system should aim to treat music artists fairly and, in that sense, include novel or “niche” artists in the recommendation lists when applicable [52]. See Section 4.2 elsewhere in this report. Fairness can be related to diversity, which should ensure that recommendations cover a wide set of items to, e.g., help music listeners explore artists that might be new to them [44]. A recommender system might enable freedom of expression as well as accessibility and inclusiveness by allowing, e.g., music artists to promote their content independent of the genre or popularity of their music [3, 45]. At the same time, recommender systems should enable users to access the content that they like and enjoy, even when their taste does not match the one of the majority of other users [17]. Transparency and trustworthiness might also be an important value for all stakeholders of a recommender system. For instance, music artists might be interested in why they are ranked at a specific position and music listeners might be interested in why a specific artist was recommended to them [50].

Furthermore, values in the area of public discourse and safety are related to a multitude of societal and human-centric aspects. Here, societal benefit goes beyond the satisfaction of individual stakeholders. As an example, a music streaming platform might be interested in fostering cultural enrichment by the recommendation of a diverse set of music [58]. This is related to the value of tradition and history, for instance, by recommending local and traditional music, which might be hard to find without the recommender system [18]. Apart from societal benefits, also the environmental sustainability might be an important value for some recommender systems stakeholders. This may involve implementing energy-efficient recommendation models within the platforms or promoting local music artists whose concerts offer the opportunity for attendance without requiring extensive travel [34]. Finally, safety is concerned with users not being exposed to recommendations of disturbing ethically questionable, or age-inappropriate content. In the case of music recommendations, this could refer to sexist or racist music tracks [35, 41].

As we mentioned earlier, the concept of “value” can be perceived as abstract, and yet, in the context of evaluation of multistakeholder recommender systems, we must be able to somehow quantify it, if the aim is to determine “goodness” for all involved.

In Section 4.4.3, we offer a theoretical construct to help navigate how to connect values to goals inhered to specific domains and (sub)sets of stakeholders involved, and how these can be operationalized and measured for assessment. Thereafter, in Section 4.4.4, we show how we take theory to practice but discuss several examples of multistakeholder recommender system applications.

The cornerstone of multistakeholder evaluation is identifying the relevant stakeholders that will be affected by or affect the recommendation process in some way, as shown in Fig. 5. The core parties in any multistakeholder evaluation are the consumers, providers and the system stakeholders behind the recommendation platform. A sensible first step is to engage with the system stakeholders and gauge their understanding of whom they are recommending to (= consumers) and where the items being recommended come from (= providers). System stakeholders, by virtue of their central role, are also most likely to have the greatest awareness of potential third-party stakeholders whose decisions may impact the operation of the recommendation platform. Commonly, third-party stakeholders would involve regulatory bodies and institutions; here, the system stakeholder’s legal department could help identify relevant regulations (e.g., related to consumer protection) and the right parties to reach out to. Finally, depending on the recommendation scenario, system stakeholders may also be helpful in identifying relevant upstream and downstream stakeholders.

Consumers (or users) have historically played (and continue to play) a central role in recommender systems evaluation. As a result, a common next step would be profiling the consumer stakeholder and the different subgroups this stakeholder category may represent. In addition to interviews with the system stakeholders, any existing market or user research on the user base of the recommendation platform could serve as a valuable foundation for identifying representative subgroups within this user base. A literature review aimed at identifying similar or related recommendation scenarios could also be helpful in identifying different user groups, especially groups that may be underrepresented in the market research for whatever reason. The system stakeholder should be able to facilitate access to these subgroups, for instance through user research panels, surveys on the website, or customer mailing lists. It is important to recruit a diverse and representative sample of consumers to represent the customer stakeholder and ensure all voices are heard in the evaluation process. Customers should be interviewed or surveyed about which values matter to them in this recommendation scenario (and their relative importance), which goals they have, and how and when they envision using the recommender system. If representative, the principle of saturation could be useful in guiding the sample size required: if additional participants do not reveal any new values, goals, or usage scenarios, then the sample should be representative of the customer stakeholder. Consumers are also a valuable source for identifying possible downstream stakeholders that are worth including in the evaluation process.

The item provider(s) are the general class of individuals or entities who create or otherwise stand behind items being recommended. Historically, they have perhaps been less well represented in recommender systems evaluation, but they play an essential role in a multi-stakeholder evaluation. The number of different individuals or entities that make up the provider stakeholder role may vary greatly between recommendation scenarios: in some cases, only a handful of entities may be providing the items to be recommended, whereas in others they may be as numerous as consumers. Similar to the customer stakeholder, the system stakeholders should be able to facilitate access to the provider stakeholders and help identify which of them are them carry the biggest weight, without losing sight of the relevant minority providers. Providers are the most valuable source for identifying possible upstream stakeholders that are worth including in the evaluation process. Again, it is important here to recruit a diverse set of representatives for this stakeholder group to ensure that their needs, values, and goals are all met in the evaluation process.

One outcome of interviewing the consumer, provider and system stakeholders should be the identification of any relevant upstream and downstream stakeholders. This could be supplemented with additional stakeholders identified through a literature review aimed at identifying similar or related recommendation scenarios.

Each of the stakeholder groups should be involved in the process of determining how best to evaluate the quality of recommendations while taking into account the values and goals of each of these stakeholder groups. Qualitative research methods, such as interviews, focus groups, surveys [29], contextual inquiry [46], and co-design [53] could all be beneficial in this process.

Once the stakeholders have been identified, the next step involves looking at the values they want to be part of the recommendation task. Stakeholders’ values are at the core of the evaluation process since they drive the modeling of the overall optimization problem. They represent high-level and abstract objectives the stakeholders wish to be satisfied via the use of the recommendation platform [35]. For instance, if the stakeholder is a music consumer a possible value is usefulness (of music experience). On the other side, for music providers, a value could be monetary reward or (societal) well-being. It is worth noticing that values may also overlap or partially compete with each other.

The elicitation of values is a fundamental step (but sometimes neglected step) as it allows the actors involved in designing the system to formulate the goals of each stakeholder involved in a multistakeholder scenario. Going back to the music consumer and provider in our hypothetical example, possible goals might be accuracy and diversity of the recommendation results for the consumer, sell as many items or services as possible, grow the number of users, sell elements over the whole catalog, protect underrepresented groups, reduce carbon footprint for the provider. Differently from values, goals can be tailored to the specific recommendation domain. A provider may set its goal as grow the number of users listening to classical music, a consumer may wish to have diverse song recommendation with respect to genre. Goals are more detailed and measurable objectives than values and they drive the design and implementation of the system through the metrics.

Specific, formal evaluation metrics provide the way to measure the extent to which the goals of various stakeholders are achieved, i.e., they are measurable proxies towards goals. For example, both consumers and providers are likely to be interested in recommendation accuracy, consumers may be further interested in item discoverability (diversity, novelty, long-tailness), providers are likely interested in increasing revenue and engagement, and the third-party stakeholders (for instance, regulators) are likely to be interested in consumer-protection-related metrics (representation, fairness, etc.).

Multiple metrics can measure the success of the same goal depending on the point of view or the aspect we want to highlight. For example, there are different metrics to measure accuracy (e.g., nDCG, MRR, or Recall), we may measure the overall number of items sold in a specific period or in a specific geographical area, the items from the long-tail and the short-head, etc. Depending on the goal, we may have metrics not targeting the overall population of users and stakeholders available in the system.

Some of the specific metrics will naturally come from the prior researchers literature in recommender systems – the reader may refer to Section 4.1 and Section 4.3 for discussions of some best practices and key metrics in recommender systems evaluation. However, there are clearly opportunities for further metric design, especially so for provider-oriented and third-party-oriented stakeholders (i.e., stakeholders that have been under-explored in recommender systems research). All the metrics must be validated by the target stakeholders (a relevant subset of the overall population is sufficient) to check if they are actually representative of their goals and if they are able to differentiate between relevant and irrelevant results. Stakeholders validating the metrics are asked to evaluate the meaningfulness of the computed results, compared to their goals. A further result of this validation process by the stakeholder can be that of identifying a priority among the metrics. Especially in this phase, one desirable characteristic of a metric is its interpretability and its propensity towards the generation of a human-readable explanation.

As the result of this step, a list of important evaluation metrics $(m_1,\mathrm{\dots },m_n)$ is enumerated, which represents the set of important considerations across multiple stakeholders that need to be taken into account as part of the multistakeholder recommender system evaluation.

Identifying the list of important evaluation metrics $(m_1,\mathrm{\dots },m_n)$, as discussed above, provides the ability to evaluate (i.e., to score) a given recommender system $R$ in a multidimensional manner; more formally, $𝐒(R)=(s_1,\mathrm{\dots },s_n)$, where $s_i$ is the performance of $R$ with respect to measure $m_i$, i.e., $s_i=m_i(R)$. Having multiple evaluation measures raises an important challenge of how determine the overall (i.e., multistakeholder, multiobjective) performance of the system [60]. In particular, given two candidate recommender systems $R_A$ and $R_B$, where each of which can be evaluated according to the stated list of metrics, $𝐒(R_A)$ and $𝐒(R_B)$, how to design a multistakeholder/multiobjective evaluation mechanism ${\prec }_M$ that allows to determine whether system $R_B$ has superior overall performance to system $R_A$, i.e., $𝐒(R_A){\prec }_M𝐒(R_B)$?

Example strategies for developing multistakeholder/multiobjective evaluation mechanisms ${\prec }_M$ include:

• $\mathrm{■}$

  Weighted (typically linear) aggregation of individual metrics [4, 32] into a single numeric score (as an overall performance), which then allows for a more straightforward comparison of candidate systems.

• $\mathrm{■}$

  Reduction of metric dimensionality by converting some of the individual metrics into constraints [59]. Constraints can be of various types, e.g., hard vs. soft constraints. Hard constraints may indicate the system performance requirements that must be satisfied, which then can be used to filter out candidate systems with inadequate performance. Soft constraints may indicate the relative importance (prioritization) of some metrics, which then can be used to rank the candidate systems accordingly.

• $\mathrm{■}$

  Determining the Pareto frontier of the multidimensional performance vectors of different candidate systems, and measuring the overall performance of a given system as its distance from the Pareto frontier [19]. One key consideration is specifying an appropriate distance metric for multidimensional performance vectors $(s_1,\mathrm{\dots },s_n)$.

• $\mathrm{■}$

  Learning ${\prec }_M$ from “ground truth” examples. This could be achieved by providing multiple examples of multidimensional performance vectors $𝐒(R_i)$ to domain experts, asking them to provide the “ground-truth” judgments regarding the overall performance, and then using machine learning techniques to learn the relationships between the individual metrics and overall performance. For instance, the domain experts could rank pairs of performance vectors at a time, $𝐒(R_A)$ and $𝐒(R_B)$, and provide a ground-truth judgment of whether $𝐒(R_A){\prec }_M𝐒(R_B)$ or $𝐒(R_B){\prec }_M𝐒(R_A)$ (or neither, $𝐒(R_A){\approx }_M𝐒(R_B)$). Learning-to-rank techniques can then be used to build a model for estimating ${\prec }_M$ from such training data.

More generally, development of multistakeholder/multiobjective evaluation mechanisms ${\prec }_M$ for recommender systems has connections to several research literatures, including multi-objective/multi-criteria optimization [13, 36], multi-criteria decision making [56] (including its various methodologies, such as data envelopment analysis [7], conjoint analysis [22], multi-attribute utility theory [26]), machine learning [40], and possibly others, which provide promising directions for further research.

Additional considerations:

• $\mathrm{■}$

  Stakeholder involvement. Most of the above approaches will likely require involvement of key stakeholders and domain experts, e.g., for determining tradeoffs between individual metrics (leading to decisions regarding relative importance weights for individual metrics or for determining which metrics should be converted to constraints), for obtaining ground-truth judgments about the overall system performance, etc. Therefore, one promising research direction is in development of participatory frameworks [30] that can enable and facilitate stakeholder groups to build algorithmic governance policies for computational decision-making and decision-support systems.

• $\mathrm{■}$

  Average vs. subgroup vs. individual performance. Important consideration: Do we evaluate systems in terms of their average performance, or should the distribution of individual performance also be taken into account [43]? For example, does higher average performance also come with much higher individual performance variance (i.e., much worse individual performance for some users/items/etc.), and, if so, what are the right trade-offs? More generally, evaluation at multiple granularities (various subgroup levels) may be of interest.

Development of evaluation mechanisms ${\prec }_M$ is important not only for the ability to perform multistakeholder/multiobjective evaluation of recommender systems, but also can also drive decisions for system design and improvement. In particular, the strategies for system design and improvement can be classified as passive or active.

Passive

These are simpler (naive) strategies of using a multistakeholder/multiobjective evaluation mechanism ${\prec }_M$ to select the most advantageous recommender system from a number of (pre-existing) system candidates $R_i$. These system candidates could possibly be generated even without any multistakeholder considerations in mind (e.g., solely using traditional accuracy-maximizing machine learning approaches) – using ${\prec }_M$ to select among these candidates would allow to incorporate desired multistakeholder considerations to some extent.

Active

These are more sophisticated strategies that attempt to integrate the multistakeholder/multiobjective evaluation mechanism ${\prec }_M$ more directly into the system design/optimization process. Two potential sub-categories of active strategies include:

• $\mathrm{■}$

  Adjust/optimize the system recommendations by incorporating ${\prec }_M$ considerations as a post-processing step (e.g., by re-ranking top-N item lists accordingly, etc.), i.e., without directly changing the learning algorithm of the underlying recommender system.

• $\mathrm{■}$

  Adjust/optimize underlying learning algorithms or designing new recommendation algorithms by incorporating ${\prec }_M$ knowledge directly into the learning process (e.g., by redesigning the loss function accordingly, etc.), so that the produced system recommendations are aligned more directly with the desired multistakeholder considerations.

The multistakeholder evaluation methodology – the identification of key stakeholders and their values/goals, the choice of most appropriate individual metrics, the development of specific multistakeholder/multiobjective evaluation mechanisms, and the use of these mechanisms to guide system design and improvement – can be viewed as an iterative process, where researchers and system designers should be aware of all the key steps and can return to iteratively refine any of them.

In reporting on multistakeholder recommendation research, we encourage researchers to include in their discussion the details of stakeholder identification and consultation, the derivation of values and goals, and the justification of metrics in terms of that work. [42] make the point that formalizations developed in addressing one problem do not necessarily transfer to other contexts. The authors were writing in the context of machine learning fairness, but multistakeholder recommendation is also highly context-specific and similar principles apply.

The first example we consider is streaming music recommendation with the key stakeholders introduced above in Fig. 6, and also included in Table 2.

We will focus here on the providers, the musical artists. There are a variety of values that such individuals might have with respect to a distribution platform like a streaming service. We concentrate here on the construct of audience: an artist will often seek to build a community of individuals who appreciate their particular musical style and contribution (connection, community and social bonding) and might, for example, come to a concert or purchase merchandise (monetary reward) in addition to listening through the streaming service.

A given musical artist might seek to understand to what extent is the recommender system helping them build an audience (use value). One can imagine the system failing in various ways. It might recommend their music to listeners interested in something else and so the recommendations are not acted upon. Or it might recommend the artist’s music only to listeners who are already fans: helping cement the audience but not necessarily building it over time. True audience building might only be evident over a long period of time (repeating habitual listening, ticket and merchandise purchases, etc.) so it will probably be necessary to create a short-term proxy for the audience-building potential of a recommender system (growth and market development).

As this is a hypothetical example, our metric here is necessarily speculative but again the aim is to illustrate a process for developing such metrics, not to solve a given evaluation problem. First, we have the problem of measuring an audience from the data available within the streaming service. Let $r$ be the musical artist and let listen count $k_u=\mathrm{\ell }(r,u,t)$ be the number of times that user $u$ listens to a track by $r$ over some standard time window $t$, perhaps one month. The audience $A_r$ can then be defined as the set of individuals for whom this count is greater than some threshold $ϵ$: $k_u>ϵ$.

As noted above, measuring audience development can have a long time scale, so a short term proxy for this quality could be to measure to what extent an artist’s music is being recommended to receptive users. There are multiple ways to determine if a user is receptive¹⁸¹⁸18For example, did the user listen to a second song by the artist, add their songs to a playlist, etc.?, but the sake of example, let us assume that we can measure the number $n$ of non-audience listeners (that is, $u\notin A_r$) who were recommended a song by $r$ and then listened to the entire song. Given that musicians have very different numbers of fans, it might make sense to normalize by the size of the artist’s existing audience $A_r$: $m_r=n/|A_r|$.

As a metric shared with individual providers, a low score on $m_r$ might raise concerns for the artist relative to the recommender system. It would mean that few new listeners are being introduced to their music. For a superstar, this might not be an issue: many people know their music already, but for an emerging artist, it could indicate that the recommender is not working as it should. A higher $m_r$ score does not necessarily mean that their audience is growing but it does mean that their music is being introduced to potential new fans. From the system stakeholder point of view, this score could also be aggregated across all providers to understand audience building across the platform’s stable of artists. Its distribution might also be interesting in terms of fairness: are some types of artists better able to build audiences on the platform than others?

In the context of educational recommender systems, our example focuses on a course content recommender system for secondary school students, possibly integrated within a learning management system (LMS) where the system could track the progress of each student and generate recommendations about what to study next. We illustrate the relationship between value-driven goals and potential measures of each stakeholder, and show how the evaluation perspective changes according to the goal in focus.

In this scenario, teachers provide the content to the recommender system platform both by selecting relevant external content (e.g. educational videos, reference books and articles) and content generated by themselves. Therefore, we define the external content generators as upstream stakeholders and teachers as provider stakeholders.

The recommender system platform generates course content recommendations for students who are consumer stakeholders and direct users of the system. Parents of the students have an indirect relationship with the generated content (e.g., in a context of recommendation of educational materials for secondary school students, parents might be interested in checking the type of material their children are using) and they are defined as downstream stakeholders. Both upstream and downstream stakeholders have an indirect relationship to the RS platform which may be relevant to identify and evaluate the value driven goals in a greater picture.

The system stakeholders are responsible of the seamless operation of the recommender system and they are obliged to ensure that the recommender system platform follows the laws and regulations stated by the school management who is among the third-party stakeholders (e.g., the recommended content should be within the corresponding curriculum for each student). Fig. 7 illustrates the multistakeholder relations, goals and potential measures in this example scenario.

Based on this example scenario, one point of evaluation of the recommender system platform could be done from the perspective of one of the goals of the consumer stakeholder. More specifically, we could evaluate the recommender system platform from the students’ perspective of passing a course, answering the question “How likely is it that a student passes a course when she follows the recommendations from the platform?” (usefulness and enjoyment, as well as personal growth). Although defined from the recommendation consumer’s perspective, other stakeholders may benefit the same evaluation. For example, the teacher could use the same measure to understand if the resources she provided to the platform are good or necessary enough (usefulness and enjoyment), and the system developers might get an understanding of the relevancy of the recommendations generated by the system beyond click through rate (use value).

Since the goal of the student is to pass the course at the end of the semester, in this example, we need to evaluate our system at the end of each semester. The system generates Top N recommendations for each student. Let’s assume that the student $S_i$ receives Top N recommendations every time she uses the system. $S_i$ may choose to accept a recommendation or do another activity on the platform. Therefore, we can measure the number of accepted recommendations by student $S_i$ throughout the semester being $n_i$. The acceptance of recommendations can be measured in different ways, but for the sake of this example, if the student clicks on any of the recommendations on the list, we assume that the recommendation has been accepted. $k_i$ being the total interaction count of $S_i$ with the system, we can calculate the proportion of the accepted recommendations to the number of whole interactions as $p_i$=$k_i$/$n_i$. Finally, at the end of the semester, we calculate the correlation between the student’s final grade in the course and $p_i$. For the sake of this example, we skip the importance of the order of the recommendations, but an evaluation metric such as normalized Discounted Cumulative Gain (nDCG) could easily be employed for this purpose. Further, the final metric that correlates the acceptance of recommendations with the student’s final score, could be calculated based on the order of the recommendations, answering the question “Is the higher the accepted recommendation on the Top N list, the better the score of the student?.”

We should note that the goals of each student may be different or we might be able to identify clusters of students who share the same goals. Therefore, the evaluation methodology could be adjusted according to not only different types of stakeholders, but the differences within one type of stakeholder. This concept of granularity has been discussed in Section 4.4.3. Similarly, different stakeholders may have different temporal requirements based on their goals. For example, the students may have a goal for the whole semester (e.g., passing the course), whereas the teachers may have goals that are needed to be evaluated in a shorter term (e.g., understanding if the recommender system platform is helpful for the students to understand the weekly topics).

The final example we consider is candidate recommendation: recommending suitable candidates for an open job position, also known as talent search or estimating person-job fit. Recruiters often play an important intermediary role in this process by assessing candidates’ qualifications, such as skills and competences, previous work experience, education level, and remuneration requirements in relation to the job [5]. Much of this candidate identification and assessment process still places a great manual burden on recruiters [38] and a recommender system that suggests relevant candidates to them to approve and supplement with their own manual searches. After shortlisting an acceptable number of candidates, each candidate will be contacted by the recruiter in a (personalized) message, highlighting their match with the job in question and inviting them to apply for the position. Such a recommendation scenario is complex and properly assessing the quality of the candidate recommendations requires involving multiple stakeholders. Fig. 8 illustrates the different stakeholders involved in this recommendation scenario and is supplemented by Table 4, which displays example goals and measures for each of the stakeholder categories.

Developing multistakeholder metrics and evaluation processes raises the question of to whom such metrics might be reported and made available. Recommender systems evaluation as discussed in this report is typically a purely internal matter of engineers or system operators understanding how the recommender is operating and seeking to improve it. It could be argued that standard summative evaluations of consumer-side outcomes are really only of interest to the system stakeholder and individual recommendation consumers can assess on their own if the system is working well for them.

The types of evaluations that we discuss here are different in that they may be of interest to parties who normally have no access to the workings of the recommender system. For example, the musical artists in our streaming example would typically have very little insight into how the recommender system is treating their content. A metric such as the “audience building” one described above could be shared with artists to help them understand what the recommender system is doing. This raises the question of what kinds of transparency the system might want to support relative to such stakeholders. We are not answering this question here, but note that provider-side transparency is very little studied in multistakeholder recommendation.

One likely reason that multistakeholder transparency has been little pursued in recommender systems research is the concern that such a facility might be used to enable undesirable adversarial behavior. A web search for the term “YouTube algorithm” yields thousands of hits from search engine optimization (SEO) firms and others giving advice to creators about how to get the algorithm to bend to their will. Additional information given to providers may enhance their ability to manipulate the algorithm in ways that are not necessarily beneficial to recommendation consumers or the platform.

Our aim in this section is to help researchers and system designers consider more holistic evaluations of recommender systems, taking multiple stakeholders into account, and examining the impact of the system across stakeholder groups. There is a separate question of governance: who, in the end, has a concrete and effective say in how a recommender system operates?¹⁹¹⁹19System governance here is different from data governance as discussed elsewhere in this report. Corporate structures often have a very concrete answer to this question, but as media scholar Nathan Schneider reminds us [49], there are other models of governance that can be and have been applied to online systems. Multistakeholder governance of recommender systems is an interesting question for future research and development.

Related to the question of governance is the question of interfaces: how do different classes of stakeholders interact with the recommender systems? There is a great deal of study of consumer-side recommendation interfaces, and a wide variety of interface designs for end users to generate and interact with recommendations. Recommender systems interfaces for other stakeholders do exist but are rarely the subject of published research. For example, YouTube provides a set of tools within their YouTube Studio application²⁰²⁰20https://studio.youtube.com to enable video creators to see some information about the viewership of their videos, but there are no detailed analytics about how the recommender system is handling their content or ways to interact with the recommender system itself.

The adversarial considerations noted above have no doubt deterred recommender system platforms from offering the kind of transparency into recommender system operations that other stakeholders might find useful. As a result, this is a highly underexplored aspect of multistakeholder recommender systems. Except for a few recent qualitative studies [8, 51], we know relatively little about provider-side experiences with recommender system interfaces.

There is nothing in this discussion that requires metrics are behavioral or off-line. [28] present a well-developed methodology for conducting user studies and interpreting them in terms of user experience. Such metrics might be exactly what is needed to understand different consumer-side aspects of a recommender system. There is no comparable methodology for understanding provider-side experiences of recommendation. It would only make sense to conduct user experience evaluation if an interface for providers exists, so this research area is downstream from the development of such interfaces.

As of today, we are used to one-shot static recommendations. Nevertheless, interactive/conversational systems are coming to stage possibly changing the way we use recommender systems. The final outcome of a conversational session depends on the way the interaction is conducted from both parties: the user (consumer) and the system (that may behave on behalf of the producer). In a multistakeholder scenario, interaction is part of the overall recommendation process and it is driven by the goals of the two actors involved in the conversation. In fact, depending on the conversation/interaction strategies, the final recommendation can be completely different and push towards the satisfaction of different goals of the involved stakeholders [24]. As a final observation, the interactive process itself may affect the satisfaction of some the stakeholders’ goals. Among others, we may cite the number of interactions to get the final recommendation [11] or the seamless perception of the interactive process [33], but these are solely consumer-side metrics. There is little development of (for example) system-oriented metrics for conversational recommendation.

All the metrics available in the literature so far look at the satisfaction of one single goal per stakeholder. This is the reason why we need aggregation techniques to find the optimal solution to the multistakeholder problem. Unfortunately, aggregation is actually a further approximation of the solution and may need further manual tuning to work properly (see Section 4.4.3.4). There could be the need for new metrics which are explicitly conceived to address the multistakeholder problem and than can be configured to satisfy the different goals selected for the problem at hand.

Consider media recommendation, a very traditional domain for recommender systems, which encompasses many popular and familiar recommendation applications, including music recommendation [38], video content recommendations [54], and news recommendation [45]. The first generation of recommender systems research in these areas focused on available datasets of ratings and assessed the quality of recommender algorithms by measuring their ability to predict these ratings or generate a better ranking list of recommendations [48]. The integration of recommender algorithms in media consumption systems such as Netflix or Spotify, and the ability to collect data beyond simple ratings further extended the range of metrics used to evaluate the systems. The current generation of media recommendation systems can track how a user responds to a recommendation, for example, by determining whether the user consumed (watched, listened to, or read) a given recommendation, partially or fully, in order to better assess the relevance of the recommendation [40].

Now that many such systems have been in operation for five, ten or even more years, it may be possible to release long-term usage datasets to facilitate evaluation that extends beyond traditional, short-term evaluations and allows for a broader impact assessment (consumption diversity, etc.). Releasing standard consumption/ratings data over multiple years will allow researchers to answer a range of intriguing broader impact questions about how recommendations change or otherwise influence consumption patterns:

• When and how much do users consume?

• Does consumption variety increase or decrease?

• Do users develop new tastes?

• Did users discover new types of content that they may otherwise have missed?

• Are users becoming more or less satisfied with their media consumption?

In several cases, media consumption systems already collect a broader set of usage data – for example, a movie recommender system can ask whether a viewer is watching alone, with kids, or with significant-other – and tracking this data over time may enable researchers to assess whether the recommender systems help to bring users to spend more time together. Some music recommender systems ask users about their current mood to better personalize recommendations [4, 43]. Releasing these data along with traditional click and rating data will help connect recommendations and watching behavior with long-term mood changes.

In longitudinal studies of recommender systems with target users, the opportunity also exists to collect an even richer range of data by asking users to periodically volunteer various forms of feedback that could be related to a broader impact. This may help better understand how recommender systems affect people’s mood, mental health, and general sense of well-being over time. An example of such longer-term studies and the data that these studies could collect is provided by the famous HomeNet project [51], which evaluated the long-term impact of Internet use to identify increased levels of loneliness and a greater sense of isolation among early Internet users.

Educational recommendation systems (including learning content recommendations and course recommenders) serve as a useful counterpoint to more traditional media recommenders [17]. Research on recommender systems in this domain is still in its earlier data-driven stage, as researchers attempt to assess performance using regular data collected before integrating recommender systems in the application context. This data-driven approach to assess the quality of personalization is typically focused on predicting learner performance when solving a specific problem or during an exam. User studies, which are natural in this domain, can also collect learner feedback on question/content difficulty, novelty, or relevance of suggested items and courses, although it does not necessarily help to assess the longer-term impact of recommendation in this domain.

However, the educational domain benefits from a much broader set of data, which could help assess several dimensions of longer-term impact. Even in relatively restricted online learning content, existing systems collect all data about user interaction with learning content, course discussions and integrated assessments. Using these data, we can assess whether a learner content recommender system has helped to make the learner more efficient (i.e., helped to gain the same level of knowledge faster) or whether it has helped the learner to achieve an improved knowledge level for a given unit of effort? Did the recommendations help reduce the number of cases where learners needed to ask questions in the forum? Or did they increase the number of cases when they answer questions from peers? Does the use of a recommender system in a prerequisite course help the learner to perform better in a future course that requires this prerequisite knowledge.

In a more traditional context, universities and colleges could collect an even broader set of data covering learners’ life beyond courses: exercise, club activity, volunteering, internship, and job placements. These data may help explore the relationship between their approach to learning and their lifestyle: does improving learning efficiency lead to a more satisfied, healthier learner, because they can spend more time exercising and relaxing? We should also be able to assess whether course recommendations helped students diversify their studies, help them become better prepared for the modern workplace, and otherwise improve their employment prospects. In fact, many universities are already collecting this type of data, augmented with various feedback from students (i.e., course feedback, internship reports), and their newly formed “analytics teams” have already gained experience using these data to assess the broader impact of major curricular innovations. This experience could be used to assess the broader impact of educational recommender systems.

An important aspect of longer-term research is the need to assemble and compare data obtained from multiple studies. A reliable evaluation of longer-term impact in a single study requires a relatively stable set of conditions over an extended period of time and reduces our ability to assess multiple research ideas or options simultaneously. Assembling results from several offline or online long-term studies may enable the research community to more reliably assess the long-term impact of multiple system design aspects. Does the specific recommender approach lead their users to enjoy a more diverse collection of artist and genres? Is it decreased or increased their overall listening? Are they more or less satisfied with how much time they spend for music listening or movie watching? Does a novel transparent interface with better user control made the user return to the system more frequently or, in contrast, pushed the users to use other systems? Which combination of algorithms and interfaces in a course recommender get their users better prepared and more satisfied with recommendations in the longer term?

In turn, the need to compare and integrate data from multiple studies makes it more important to agree on the set of long-term focused data to be collected and a set of long-term impact factors to measure. Moreover, in order to enable this type of meta-analysis, the recommender systems community will need to evolve its approach to evaluation to adopt a level of experimental rigour and reporting standards that facilitate such opportunities; see Section 4.5.5.

Part of the challenge of conducting long-term research (whether retrospective analysis of data sets or experimental user studies) is that new ideas and phenomena arise for which the collected data or experiment design are inadequate. For example, a dataset collected to study the quality of recommendations may not have captured data that would allow assessing the diversity or unexpectedness of those recommendations. An experiment looking at the effects of different recommender algorithms or interfaces on consumption may not have baseline data on user attitude towards the recommender or brand. Accordingly, there is an increasing need to develop standard suites of metrics and data sets to support such long-term impact research.

It is beyond the scope of this section to specify the contents of such a standard – rather we make the case that researchers in the field should promulgate and evolve such standards with a goal of converging to a relatively comprehensive set (and note that several other sections of this article, including the section immediately following, propose partial solutions). We suggest that some factors to consider include:

1. 1.

   The challenge of identifiability of users in the context of such data, and therefore the possible need for explicit informed consent. (Consider for example [7].)

2. 2.

   The desirability of preserving not only user interactions, but also the system prompts that lead to those interactions (e.g., recording displayed recommendation sets).

3. 3.

   Recording a set of interval metrics on a regular schedule (e.g., periodic logs of consumption properties, recommendation properties, logins, etc. for the past week). (Consider [49] or [14].)

4. 4.

   Developing a suite of general attitudinal and beyond-system behavioral survey questions that can be administered regularly (subject to the appropriateness within the system context).

5. 5.

   If you want to measure change, do not change the measure. In order to avoid issues with inconsistency over time, statistical validity, reliability across waves, bias introduction, and introduction of confounds, the metrics used should be well-thought upfront and not changed throughout the long duration of the study.

Systems are learning about preferences and behavior while interacting with users over time and thus have long-term impact. Companies optimize their recommender systems for business metrics, but they do not necessarily account for other impacts (e.g. Netflix optimizes for hours of viewing but is not aware if these hours are quality time or addictive binge-watching [18]). Binge-watching, for example, has been studied in psychology and has been related to mood regulation [73]. Hence, understanding how mood regulation works can inform the choice of metrics to be used for measuring the impact of a recommender system.

The choice of metrics is domain-dependent. Here we provide a non-exhaustive set of theoretical concepts relevant to the impact of recommender systems on both individual users and society. The concrete metrics used need to be adjusted to the specific domain. These theories primarily draw from psychology and other social sciences. Furthermore, there are additional relevant theories from economic, cultural, behavioral, and various other fields that can also be considered to fully understand and measure the long-term impact of recommender systems.

Individual user behavior can be better understood if we recognize that users differ substantially in their personal characteristics. Some of these characteristics are hard to change (e.g., personality) while some others can be affected by the exposure to recommender systems. For example, the user’s level of expertise in a domain, their personality [78], their decision-making style [6], and their need for autonomy/independence. As an example for expertise, the Music Sophistication Index (MSI) [59] has shown to be a substantial factor in understanding individual differences in user interactions with the music recommender [27]. On a more specific level, their momentary behavior will be affected by their attitudes, values, and beliefs: these are relatively stable in the short term but might drift over the course of time, potentially influenced by the interaction with a recommender system. There are several models describing how these aspects influence current behavior, such as the Theory of Planned Behavior [2]. In light of our perspective on long-term evaluation, this suggests that measuring attitudes, beliefs and values on a regular basis might help us better understand what is driving users’ long-term interactions with a recommender system. Similarly, users’ mood [79] or mental well-being might fluctuate over time and play an important role in their interactions, and any measures that might capture these implicit or explicitly [84, 81, 83] would be helpful in better understanding long-term interactions.

A special psychological construct relevant for recommender systems research is the concept of user preferences. Recommender systems take user preferences as somewhat stable and measurable, but psychological research has shown that people often do not really know what they like and construct their preferences while making decisions [11, 34, 25]. A recommender system thus also allows people to better understand their preferences, and as the recommender system learns over time, the users might also learn about their preferences from the interaction with the system. Moreover, research makes the distinction between actual (current) preferences versus more ideal (value-based) preferences [54, 46], which is directly related to the distinction between short-term and long-term goals, which we will discuss in the next section.

Social theories are crucial in understanding the broader implications of recommender systems on collective behavior and societal structures. These theories can, for example, help understanding how public sentiment, political polarization, and education and awareness might be impacted by recommender systems. Social theories and their analytical frameworks can also help study the effects on societal tolerance, diversity, and potentially economic inequalities or environmental sustainability. Concepts, such as cultural identity, social capital, and civic engagement can also be examined, providing insights into how recommender systems can shape or impact social norms.

Some impact metrics can be computed from logs (e.g., hours of reading news), while others require more specialized instruments (e.g., measuring user sentiment toward a political issue in news recommender systems). These instruments, such as lengthy questionnaires, can be costly and time-consuming to administer. Therefore, a balance between accuracy and scalability is essential. One option is to measure just a small sample of users and build a predictive model for the remainder of the population under study. For example, asking a couple of hundreds users to gather ground truth labels and then training a predictive model from user behaviour logs.

To measure how a news recommender system affects political polarization, one could sample the current sentiment of the society on a regular basis (e.g., weekly) over a longer period of time. The theories that inform the choice of the metric could be, for example, the social identity theory [85] (in-group favoritism and out-group hostility) and the cognitive dissonance theory [28] (reject or rationalize information that contradicts people’s beliefs). These theories could lead to a choice of metrics, such as measuring the sentiment of people towards in-group and out-group generated posts (can be computed from digital traces in social media) or a questionnaire-based instrument that measures the cognitive dissonance of users when/after being exposed to a certain news item.

Psychology has studied extensively the conflict between short-term needs and desires and long-term aspirations and goals and how to help people overcome their short-term desires to focus on the long term. Models of behavioral change talk about different stages in which users go from awareness to motivation to change to action (e.g., the transtheoretical model [69]). Are we able to capture such stages in the data and develop metrics for them?

An effective approach to achieve long-term goals is to break the long-term goal into smaller and attainable short-term goals. Such short-term goals have a prospect towards attaining the larger long-term goal, but most recommender systems do not have a notion of a long-term goal being behind the interaction / behavior of the user. Some exceptions are Rasch-based recommender systems [75, 72] and other approaches [82, 10], which models the user’s ability and item difficulty, allowing the system to recommend items that are within their ability, thus allowing for smaller short-term goals (I can run 5km now) to be achievable and to develop towards attaining long-term goals (I want to run a full marathon). In any case, recommender systems might need to be aware of such long-term goals, and it is quite likely that we cannot learn about such goals by just observing user behavior with the system. A conversational approach between the system and the user might be needed to make sure that what the recommender system is learning about the user reflects the user’s longer-term goals. But how should the system communicate to the user what it learned and based on what metrics? There is an opportunity to develop algorithms that take into account the balance between optimizing for short-term (attainable) goals while still being on track to achieve users’ long term goals. What metrics would we need to optimize for both?

There is an inherent temporal aspect in distinction between long- and short-term goals. Long-term goals are by definition more into the future, though they might influence current (short-term) decisions. Research on inter-temporal choice shows that we devalue future gains and prefer immediate rewards over delayed ones and that users need to have awareness of their future goals to overcome this, for example by changing their perspective. For example, we can prevent people from procrastinating by making the goals explicit, making people think more about their future selves, or reserving their mental queries [44]. How can we build recommender algorithms that support such strategies, that can recognize items that satisfy long- and short-term goals and based on what metrics?

Better measurement and modeling of long-term interactions between users and recommender systems also offer opportunities for cross-fertilization between social science disciplines and recommender systems research. For example, social science research in behavioral change has shown to have limited practical impact because often studies are designed for understanding and theorizing rather than really helping users move forward. Actual intervention studies are typically done on a much smaller scale compared with large-scale recommender systems experiments. Moreover, these studies typically do not use highly personalized interventions, as they lack the (long-term) behavioral data and algorithm expertise to do so. The outcomes from better long-term recommender systems evaluation studies can inform future intervention studies. Furthermore, recommender systems researchers could team up with domain experts to build multidisciplinary teams to combine the strengths of both worlds.

Typically, recommender systems are used in domains where there is an abundance of products or items, helping users discover new content that they might not have found otherwise. This way, not only do the most popular items get visibility, but the long tail of less popular items and niche content can also find its way to its specific audience.

Ideally, recommender systems are beneficial for the long-term goals of its providers, improving overall engagement, retention, and increased revenue for commercial providers [23]. In practice, recommender systems are often optimized using several short-term metrics focusing on immediate user interactions and engagement. Unfortunately, such short-term focus does not necessarily correspond with long-term benefits or might even become harmful for it. Here are some example metrics commonly used.

• $\mathrm{■}$

  Clickthrough Rate (CTR): One of the most commonly used metrics that measures the percentage of recommended items that are clicked by users. It is a direct indicator of how engaging or relevant the recommendations are perceived to be. Systems optimized for CTR, however, can suffer from clickbait. These are recommended items that typically have sensational or misleading titles or images designed to attract clicks. While such content might generate high immediate engagement, it is typically low in quality and does not provide lasting value to users. This can significantly degrade the overall and long-term user experience on the platform. Moreover, such clickbait is likely to result in a feedback loop where the recommender system will put even more emphasis on such sensational content, limiting the diversity of content that users are exposed to.

• $\mathrm{■}$

  Conversion Rate and Monetary Value: This metric tracks the percentage of recommendations that lead to a desired action, such as making a purchase, signing up for a service, or the economic value that it brings. Conversion rates are typically used for e-commerce and service-oriented platforms. Recommender systems optimized for conversion or monetary value might put too much focus on high-price or high-margin items, which might not be the most optimal choice for long-term monetary value. For one, this metric does not take the post-purchase experience into account and users could lose trust in the system in the long term.

• $\mathrm{■}$

  Immediate User Feedback: This can include thumbs up/down, star ratings, shares, comments, or any other form of quick feedback that users provide after interacting with recommended items.

• $\mathrm{■}$

  Session Duration: This measures the total time a user spends on the platform during a single session. Longer session durations typically indicate that the recommendations are engaging users effectively. Although session duration is very closely related to CTR, it could improve the quality of the recommended content, as CTR alone does not capture the time a user eventually spends on the clicked item.

• $\mathrm{■}$

  Bounce Rate: The percentage of users who leave the platform after viewing a single recommended item. A lower bounce rate suggests that users are finding value in the recommendations and choosing to explore more content.

• $\mathrm{■}$

  Item Coverage: This measures the proportion of the catalog that is recommended over a period of time. Higher item coverage indicates that the system is leveraging a broader range of available content, which can be beneficial for both users and content providers. This is one of the main strengths of recommender systems, activating the long-tail of the catalog and matching niche items to the users interested in it [41].

• $\mathrm{■}$

  Hit Rate: The proportion of times the recommended item is the one that the user interacts with. This is a straightforward measure of the accuracy of the recommendations.

These metrics are essential for understanding how well a recommender system performs in the short term and are often used to guide iterative improvements and A/B testing. However, while these metrics are useful for immediate optimization, they all share the risk of reduced content diversity, and pressure on content creators and vendors to optimize their offering solely to boost the used metric.

Apart from content diversity, there are significant risks of several other long term effects that need to be considered when deploying a recommender system.

• $\mathrm{■}$

  Filter Bubbles, Echo Chambers, and Polarization: Recommender systems can create filter bubbles, where users are only exposed to content that reinforces their existing beliefs. This can lead to echo chambers, reducing exposure to diverse perspectives and potentially fostering polarization [57, 5, 65].

• $\mathrm{■}$

  Addiction and Overuse: Systems optimized for short-term engagement can encourage excessive use, leading to addiction. This is particularly concerning on social media and video streaming platforms, where the continuous feed of recommended content can lead to unhealthy consumption patterns.

• $\mathrm{■}$

  Bias Amplification: Recommender systems can amplify existing biases present in the data. For example, they may disproportionately recommend content from certain demographic groups or types of content, reinforcing societal biases and inequalities.

• $\mathrm{■}$

  Privacy Concerns: Long-term data collection for improving recommendations can raise significant privacy concerns. Users may become uncomfortable with the amount of data being collected about their preferences and behaviors over time.

• $\mathrm{■}$

  Content creators: Recommender systems can skew visibility and revenue opportunities towards already popular content creators, making it harder for new or niche creators to gain traction (see also Section 4.4. This can lead to a lack of diversity in the available content and reduce the overall variety and innovation within the content ecosystem. This situation limits the exposure of different types of content and discourages new creators from participating, which negatively affects the richness and dynamism of the platform.

• $\mathrm{■}$

  User Manipulation: By optimizing for engagement or sales, recommender systems might manipulate users into behaviors that are not in their best interest, such as overspending or engaging with misleading information, or even causing emotional impacts [80].

• $\mathrm{■}$

  Reduced Serendipity: Over time, users may be less likely to encounter unexpected or novel content that could enrich their experience, leading to a more monotonous and less stimulating interaction with the platform.

Given the problems stated in the previous section, we suggest taking a more ambitious approach to evaluating recommender systems in practice. In this section, we discuss what more ambitious recommender systems could entail and in the next section, we go into more detail of how to proceed with implementing such goals in practice.

In this pursuit, those who are responsible for the roadmap of these systems should recognize the relevance of these long-term objectives. Within the domain of science and technology studies, the social construction of technology (SCOT) emphasizes the notion that technological systems, such as recommender systems, are influenced not solely by technical elements but also by social processes and human decision-making [47]. In the context of recommender systems, this perspective highlights that these algorithms are designed and implemented based on specific choices made by developers, product managers, and other stakeholders [74].

While public discourse often portrays (the impact of) recommender systems as inherently “bad” or problematic, for example, the highly popularized filter bubble hypothesis by [65], this SCOT lens reminds us that the perceived issues or biases in these systems stem from the underlying human decisions and values embedded in their design and development. In other words, recommender systems do not operate in a vacuum but reflect the priorities, assumptions, and trade-offs made by the individuals and organizations responsible for their design and implementation.

However, this does not imply that there could not be any (positive or negative) unintended consequences. For example, as pointed out by [87]: “a recommender system designed to serve its customers may unintendedly (and systematically) contribute to filter bubbles and echo chambers […] although that was never intended by its designers.”

Apart from these unintended consequences, a goal-oriented recommender design in practice is informed by answers to the questions:

• “What kind of recommender systems do we want to develop?”

• “What kind of objectives do we want to achieve?”

This is where it gets challenging. To answer these questions, one should have a thorough understanding of both the specific domain and the various purposes that may or may not be achieved by using recommender systems.

In most cases, stakeholders either are domain experts or recommender systems experts. These domain experts have the knowledge to define the long-term goals, which should then inform the evaluation criteria of the recommender system. In practice, this translation from “goals” to “metrics” [35] is a joint effort by domain experts and recommender systems engineers, which is mediated by product owners who facilitate the interaction between these stakeholder groups. In this effort, it is essential that each of these stakeholders is informed about the range of possible long-term goals of these systems. This is to ensure that they are not constrained by the narrow range of objectives that have been dominant thus far in recommender applications in practice, as previously outlined in this section.

To build a broader, or more ambitious, understanding of such long-term recommender goals, practitioners may rely on examples discussed earlier in this report or recent surveys [37, 36]. Additionally, a north star in this context could be the UNESCO four core values (Ethics of Artificial intelligence) [86] (See also the discussion of values and goals in Section 4.4.2):

• $\mathrm{■}$

  Human rights and human dignity – Respect, protection, and promotion of human rights and fundamental freedoms and human dignity

• $\mathrm{■}$

  Living in peaceful – just, and interconnected societies

• $\mathrm{■}$

  Ensuring diversity and inclusiveness

• $\mathrm{■}$

  Environment and ecosystem flourishing

While these should be the guiding stars for all AI technologies, optimizing for or even understanding how one system can impact these values might not be easy. Instead, we propose to focus on more specific goals in their respective domain. In the following section, we will look at more specific cases.

A currently open question is: What are the effects of the explosion in social media usage and the new online lifestyle? However, most research [32] seems to indicate that the current level of addiction should be limited, especially for young people but also for the population in general. It is, therefore, prudent that recommender researchers and engineers start thinking about the long-term impact of their systems.

Monitoring how the users are progressing towards the long-term goal could be done simply by asking the user. For example, Duolingo has done extensive work to understand how to measure long-term effects with short-term metrics [30] and adds quizzes and review exercises to understand the progress of its learners better [68].

Another approach could be to request domain experts to define measurable proxy metrics in a shorter time frame, enabling the engineers to optimize the system accordingly. For example, focusing on customer lifetime value could simply be reduced to optimizing users’ chances of returning to the platform [30].

There is often a discrepancy between ideal and actual preferences among users and providers of recommender systems. For instance, while the nutritional benefits of broccoli are well known, recommending it may diminish trust in the system because users may prefer crisps instead. To ensure long-term effectiveness, recommender systems should prioritize optimizing for long-term goals. In the aforementioned example, instead of consistently suggesting crisps, an optimization strategy could be to gradually increase the instances where the user selects the ideal choice without causing them to abandon the platform.

Similarly, one might also consider the addiction-like problems that users experience with social media platforms, where the platforms’ optimization criterion is to keep users engaged as frequently and extensively as possible [32]. Instead, these social media platforms could set goals to encourage and support physical meetings and events. Could it even be advantageous for these platforms to have users spend shorter but more focused time on their platforms? This approach could provide a more effective platform for marketing, as it might engage focused users rather than relying on the large percentage of mistaken clicks that currently inflate the platforms’ metrics. The success of such an approach could be measured with check-in-like features that allow users to demonstrate that they met in person.

Social media has also become many people’s main news source, giving a unique chance to provide complete news coverage not only covering many diverse stories but also with opposing views of stories (from different newspapers), ensuring that a user gets exposed to a wide set of topics. While it might not be possible to provide opposing views of stories for a single newspaper, they could still adopt similar goals. Editors should define their overall coverage goals, and a short-term metric would be to optimize so that users would get as complete coverage as possible.

One of the biggest challenges for businesses today is to define the tests that will allow them to understand long-term impacts better. In practice, the first task would be to define these long-term goals and align those with the stakeholders. The long-term goals should then be translated into Overall Evaluation Criterions (OEC) to ensure they are measurable [50]. As these long-term goals often require measurements across a longer period of time, one might be tempted to suggest that online tests should simply run for longer. However, often, that obstructs other goals of stakeholders, and even if allowed, this does not come without its pitfall either [16].

Another more feasible approach is to create a set of short-term proxy metrics that will enable performance measurement towards long-term goals but in short-term feedback loops. Using proxy metrics is not always straightforward, as described by [63]. It is, therefore, important to capture the proxy metrics and compare them with the long-term goals at intervals and monitor the overall system to ensure that using these metrics won’t hurt the system.

Research shows that diversifying recommendations increases the user experience. This is not the best strategy when optimizing for short-term goals, but by diversification, the system learns new user preferences [82], and even if it might result in lower short-term performance, it will be a good investment for the system’s long-term performance.

Lastly, most metrics reported only look at the positive increase, but this can very well hurt minorities, as they might be hurt badly by changes, which might still go into production because an A/B test is considered successful. Similarly, another metric seldom considered is the churn rate of users who receive recommendations that affect them to the point that they leave and never return. Most short-term metrics optimize for positive reactions, while bad recommendations are never tracked and monitored. Not doing this will eventually lead to a loss of users. Returning to the examples of broccoli vs. crisps, it is important to show recommendations for broccoli to encourage healthy behavior, but not to the level that makes users not return to the platform.

This section presents few examples from the recommender systems literature wherein long-term and longitudinal aspects have been considered when assessing the impact of the recommendations. First, there are presented works employing two different methodological approaches, simulation-based environments and longitudinal user studies. Then, examples of studies which consider specific scenarios emerging from continuous interactions with recommender systems are described: feedback loops, impact of content diversity, and rabbit holes.

Simulation based-environments, such as Agent-based Modelling (ABM) has been employed to explore the long-term impact of recommender systems on various aspects [1]. [88] focus their efforts on simulating users’ consumption strategies, demonstrating how, through reliance on recommendations, individuals might inadvertently contribute to a decrease in overall variety over the long term. [89] utilize ABM to investigate the impact of preference bias – the distortion in users’ self-reported ratings resulting from recommendations – on the effectiveness of recommender systems. Specifically, they demonstrate how the system’s performance can be adversely affected by the introduction of user-rating-induced bias, potentially compromising the overall variety of recommended items. [39] concentrate on the analysis of recommendation techniques through an iterative approach, where users are presumed to engage with a particular portion of the recommended items. They demonstrate that, in terms of recommendation dispersion and coverage, several systems evaluated exhibit an increased concentration over time. Employing a similar methodology but focusing on session-based recommender systems, [26] uncover similar findings concerning spread and coverage.

Longitudinal user studies are quite rare in recommender system research, due to the large amount of resources and time needed in order to gather data on the interactions between users and systems. An eight-week longitudinal study between subjects has been conducted by [15], designing an app where participants received personalized recommendations for physical activities and guidance to minimize sedentary behavior. In the work by [33], the impact of personalized recommender systems is examined. The system provides visual feedback and recommendations based on individual dietary behavior, phenotype, and preferences. By employing quantitative and qualitative measures over a 2-3 month period, the study demonstrates that the system positively impacts nutritional behavior as measured by the optimal intake of each nutrient. In the music field, [53] present a longitudinal study, focusing on users’ exploration behavior and change in behavior after employing a music genre exploration tool for four sessions across six weeks. [67] present the outcomes of a 12-week longitudinal user study, involving participants who received daily music diversified recommendations. By analyzing their explicit and implicit feedback, it is demonstrated that exposure to particular levels of music recommendation diversity in the long-term may impact listeners’ attitudes.

The decisions made by recommender systems can shape user beliefs and preferences, which subsequently impact the feedback the system receives, thereby establishing a long-term feedback loop. [42] provide a theoretical analysis of the relationship between feedback loops, echo chambers, and filter bubbles. [13], through the simulation of various user engagement models with recommender systems, demonstrate the influence of feedback loops on the homogenization of users’ behaviors. [56] design a model to iteratively analyze the feedback loop, showing how it may be responsible for a decline in aggregate diversity. Focusing on the long-term impact on exposure, [21] discuss how recommenders may exacerbate the rich-get-richer effect, strengthening exposure inequalities. Challenges derived from the presence of feedback loop are also common in industry settings, and [85] show how to address long-term feedback loop emerging issues by using an offline evaluation framework.

Content diversity has been at the center of attention of numerous studies due to its relationships with filter bubbles and echo chambers, among the undesired long-term impacts most researched in the recommender system literature [57]. [8] employ numerical simulations to model user decision-making processes, offering an explanation for the findings of a prior study by [60] on the impact of recommender systems on content diversity. In the original work, the authors found that users engaging with the provided recommendations consumed more diverse content compared to those who did not. [8] corroborate these results, but they also report an increase in user homogeneity – a decrease in aggregate diversity. Similar results are presented also by [3], who observe a connection between recommendations and long-term reduction of diversity. The narrowing of the range of content to which users are exposed is also relevant when the pathways that recommender systems define lead to the consumption of polarized content – eventually contributing to user radicalization – creating what are nowadays commonly referred to as rabbit holes [64]. Under this lens, YouTube recommendations are examined in the work by [70] and [22] in the context of user radicalization.

The goal of experimental and empirical research is to contribute new knowledge that future researchers and practitioners can use and build on with confidence. Fields of research generally rely on two mechanisms for ensuring that proposed contributions deserve that confidence:

• $\mathrm{■}$

  Peer review – the evaluation of work by other experts in the field

• $\mathrm{■}$

  Standards – the adoption of practices viewed as best practices for research

Consider, for example, a medical researcher who wants to test whether high doses of vitamin C affect the incidence of influenza among people who take it. Standards exist for clinical trials to constrain the methods (e.g., protocol development, funding approval, protocol registration, a double-anonymous, placebo-controlled, random assignment trial) and the parameters of the experiment (e.g., through power analysis and pre-determined significant effect sizes) [58]. A peer review process would likely be applied twice – once beforehand of the study design (either as part of a funding decision or as part of human subjects ethics review), and then again afterward on the final manuscript. Even then, standards of publication would ideally ensure that sufficient detail be included in the publication to support both replication and later meta-analysis for assessing the impact of previous research studies.

In recommender systems, like many other computer science-related fields, in contrast to the medical domain mentioned above, our mechanisms for ensuring confidence in results have been more limited. We have peer review prior to publication, and have numerous best practice guidelines published (e.g., [14, 48, 49, 38]). We also have ACM guidelines for conducting studies with human subjects and the need for institutional IRB. But there are no accepted or agreed on standards for reporting results, no pre-review of experiments (as a way to demonstrate that the hypothesis, approach and analysis were planned in advance and not shaped by data as they emerged), and rarely any mandate to authors or reviewers to reference best-practice guidelines as part of publications and their review. We should note that recommender systems research benefited significantly from this flexibility in its early years. Exploratory work like the early recommender systems implementations needs rapid exposure more than iterative refinement. Indeed, there still is and always will be highly exploratory new work. But we believe that the majority of research in the field is incrementally advancing the science and practice of recommender systems and would therefore benefit from increased focus on rigor.

In this section, we make recommendations aimed at improving research rigor and the confidence with which research results can be applied. While some of these recommendations are general and can apply to any empirical or experimental research, we focus primarily on high-cost research (such as longer-term and large-scale experiments, but also computationally expensive multi-dataset experiments and simulations) where peer review of research design may help address study design problems in a timely and cost-effective manner.

In the Registered Reports track, submission approval primarily emphasizes the significance of the research questions and soundness of the research protocol, in alignment with open science principles, rather than the study results. This model is different from the traditional model of publication in most conferences and journals within the recommender system and computer science fields. Therefore, it might require community efforts and time to evolve this initiative to enhance research rigor. By looking at Registered Reports initiatives reflecting in other fields [12], we might foresee challenges in successfully implementing this new model. At the forefront of our considerations are the following:

Enhance awareness of open science in the whole community

In our research community, as well as the broader computer science research community, only a few journals or conferences offer Registered Reports. For instance, in software engineering research, Registered Reports were first introduced in 2020 at the International Conference on Mining Software Repositories [19]. It might take some time for researchers to receive the necessary training and education to raise awareness and understanding of open science principles and to understand the benefits of such initiatives. We hope this proposed track can serve as a starting point for the entire community, fostering collective efforts to improve research rigor.

Motivate submission of Registered Reports earlier enough

The most obvious barrier to Registered Reports is time. Researchers need to wait for peer review feedback and approval before conducting their study and collecting data, which can be challenging for those on short-term contracts or within short funding cycles. Additionally, students and early-career researchers may need to acquire the necessary skills and knowledge to write a registered report, potentially causing delays in submission and then study execution. While registered reports could benefit our research in the long run, we also need to consider how to better support researchers at different stages in submitting their registered reports early enough to address these practical challenges.

Ensure effective peer-review

Another significant challenge is to ensure that the peer review process functions well, as the benefits of this Registered Reports initiative highly depend on it. This requires both the careful selection of qualified reviewers and successfully engaging them in the review process. Currently, peer reviewers in the community are not always trained to engage deeply with the experimental design and study procedure of the submissions they reviewed. Additional training for reviewers may be necessary for this new model, which might pose further challenges for editors in finding suitable reviewers. To help address this issues, we may consider providing some form of credit for reviewers who offer substantial formative feedback to authors in this model.