Reviewer No. 2: Old and New Problems in Peer Review

Currently most conferences do little to encourage good reviewer behavior. The ⁢ACL conferences recently created a policy to increase the number of reviewers and chairs nominated for awards to 1-1.5%⁵⁵5https://2023.aclweb.org/program/best_reviewers/, but this is still not enough: the chance to get such an award is still relatively small and not worth extra effort on the part of the reviewer. One suggestion was that if the awards are given to as much as 50% reviewers, this would reverse the incentive structure: not getting the award would create a negative social signal.

The current award offered to the outstanding reviewers at *ACL conferences is either free conference attendance as a virtual participant, or a discount on the in-person attendance. This does not necessarily reflect the needs of the reviewers, some of whom come from wealthy industry labs and do not need the monetary incentive. A survey could be organized to establish what other kinds of incentives could be useful. Some other reward ideas proposed in the discussion included:

• $\mathrm{■}$

  Sharing reviewer history to ORCID, given that it’s possible to create generic service records there (e.g. conference names, number of papers reviewed, reviewer awards);

• $\mathrm{■}$

  10% conference discount to 50% reviewers? (PeerJ case)

• $\mathrm{■}$

  Sharing generic reviewer history with potential employers (e.g., lab leaders looking for PhD students/postdocs), as it provides a useful signal about reliability and commitment;

• $\mathrm{■}$

  Likewise, area chairs could find it useful in grant applications if the area and statistics of their work could be shared to showcase community leadership in their research area (e.g. “outstanding area chair for question answering track, ACL 2024”);

• $\mathrm{■}$

  Various certificates showing different levels of achievement (reviewer certificate, outstanding reviewer certificate);

• $\mathrm{■}$

  A “star” system for reviewers where people can gain a star for good reviewer behavior (e.g., number of reviews, on time, high-quality, detailed reviews, emergency reviews), and feedback can be given to reviewer for moving up in the star scoring.

Most conferences, with which the participants were familiar, use unstructured or semi-structured review forms, with questions like “summary”, “strengths” and “weaknesses”. An alternative is to have structured review forms where reviewers are asked to evaluate various aspects separately. The group discussed evidence from Elsevier studies on the use of structured review forms [38], which increase agreement between reviewers, and decrease the cognitive load (allowing them to comment on the aspects of the paper for which they have expertise). From the perspective of program chairs, structured review forms can allow the chairs to better understand which parts of the paper have been reviewed more reliably, and mitigate issues like commensuration bias [32, 43]. Such forms could include concrete questions relevant to a specific paper type: e.g. “Is the statistical analysis sound?”, with the answer option “I’m not an expert on this”.

Structured review forms allow for better control to remove subjective categories in the review where biases from author identities are easier to creep in. If reviewers focus on the technical correctness of the work, these aspects might alleviate the problem of non double blind reviewing.

At present, most conferences are organized on one-off basis, with most program chairs not having access even to reviewer history from the previous editions of the same conference, much less across venues. However, each cycle generates much useful information (late reviews, low-quality reviews, outstanding reviews etc.), and simply being able to track and use this information would probably help a lot in increasing the review quality. But there is no structure in place to maintain and organize this information. ACL Rolling Review could theoretically perform this for NLP community, as it already performs some long-term data collection [11], but there needs to be a broad mandate of storing and using this information for the organizational purposes.

It is not clear what should happen to the “bad” reviewers: simply de-prioritizing them in assignment process is not a negative incentive, since in practice it just means less work for them. Conversely, the “good” reviewers should not be rewarded by simply having more work assigned to them.

The group also discussed the privacy vs equity issue: peer review data is highly confidential and should not be disclosed without reviewer consent, especially since authors are often tempted to publicly bash their reviewers. At the same time, in some cases the privacy considerations protect the wrongdoers. One possibility to introduce reviewer consent to this process is to have reviewer agreements include an optional checkbox for granting the authors the right to use reviews however they choose, including for public discussion.

Finally, the group discussed the possibility of having a single reviewer profile within a system with an institutional memory, such as ARR, that would list various items from reviewer history, so as to visualize their impact on the community and highlight the fact that their reviewing record is tracked. This profile could include the following information: how many reviews they did, for what tracks, how many were late, how many were emergency reviews, what were the outcomes for the reviewed papers, how many best paper nominations (and the outcomes for those papers), ratings distribution for this reviewer vs conference mean, length of reviews vs mean, number of discussions vs mean, any feedback notes from area chairs, number and issue types flagged by the authors.

Most of the current *ACL conferences have the more prestigious main track publication (e.g. proceedings of ACL), and a less-prestigious Findings https://2020.emnlp.org/blog/2020-04-19-findings-of-emnlp publication with a higher acceptance rate (usually 30-40%). The current workflow for peer review through ACL Rolling Review is that acceptance decisions are decoupled from review process, and are done independently by senior area chairs, sometimes months after the reviews were finished. This is unsatisfactory for the authors, who of course want faster decisions and not just reviews.

The group discussed the possibility of having ACL Rolling Review provide at least Findings decisions, which can be made purely on the ground of the judgement of technical soundness of the paper. Once such a decision was made, a Findings publication is guaranteed. Then the authors can still commit the paper to be considered for the main track publication. If the paper was rejected from Findings, it needs to go through the revise-and-resubmit process.

Based on the above discussion, the seminar participants developed a proposal to the ACL executive board to establish the ACL Peer Review Committee: a working group dedicated to the development of peer review policies across ACL venues. This group would consider and help to develop proposals relevant to the peer review process, which originate either from ACL venues or the community. It would also monitor the implementation of any proposals, and ensure that the venues implementing them would consistently report on the results of any changes.

The proposal was approved, and the committee will be formally established in 2024. This committee will have a broad mandate to analyze internal peer review data from ACL venues for the purposes of developing evidence-based policies and improving the peer review process (but not for independent research by the committee members). For the sake of transparency, any results of such analysis will be made available as public reports.

While this committee will serve only ACL community, it is a good test case, since it has a lot of major conferences and already possesses the infrastructure for shared organization and institutional memory between them (ACL Rolling Review). The successful practices from this initiative could be shared with other communities.

The conferences often start recruiting from the reviewer lists from past conferences, which means that a biased sample could keep being reused. Extra reviewers could be brought in through the networks of the chairs, which could also contribute to the bias. In ACL’23, there was a vast imbalance between the number of submissions and reviews contributed by Chinese researchers [54].

Perhaps the process should be more often, with the reviewers recruited through an open call and self-nomination via a sign-up form. Community groups such as Widening NLP could also help. To better estimate the extent of the problem, a conference registration form could include a question about reviewing (e.g. Have you published here in the past 10 years? Have you reviewed here? If not, why?)

For the specific China under-representation issue, the recommendation is to make sure that there are enough area chairs who are from China and based in China, and to ask them to help recruit widely from their networks. It would also help to connect with the National Science Foundation, there is an identifier system that is widely used within China.

ACL used to have an embargo on posting papers on ArXiv prior to submitting to ARR or ACL conferences; but the policy recently changed to lift all restrictions.⁶⁶6https://www.aclweb.org/portal/content/report-acl-committee-anonymity-policy It is still considered an integrity violation to search for papers one is reviewing, although some reviewers might use it a check for plagiarism. This change of policy necessitates discussion of how we can protect double-blind review, given that single-blind review is known to be influenced by demographic features associated with authors, such as country of affiliation, lab affiliation, fame, seniority etc. [46, 68, 39, 22, 59]. Among the possible ways to remedy this situation the seminar participants discussed the following:

• $\mathrm{■}$

  Peer review platforms could try to automate the checks for plagiarism, undisclosed preprints, previous publications etc. Then the reviewers could be told that they do not need to look for this. However, it is likely that many would still deliberately deanonymize the submissions [50].

• $\mathrm{■}$

  During paper-reviewer matching, the reviewers who disclosed the knowledge of a submission could be de-prioritized. However, the reviewers would have to input this information for all submissions that they are qualified to review. This also provides an extra opportunity to collusion rings [70, 34, 26], whose members could pretend to recognize work outside of the ring to increase the chances of assignments to each others’ papers.

• $\mathrm{■}$

  The scores from reviewers who recognized the submission could be visually distinguished in the review reports presented to chairs, to help them downweight such scores (as they are potentially unreliable).

• $\mathrm{■}$

  “Confidence” score is too ambiguous, it could be interpreted as confidence about impact, novelty etc. It should be worded as confidence in the assessment of the technical aspects of the paper. Maybe this could help the reviewers to calibrate their assessments better even in a single-blind situation.

A related issue is equity: peer-review work needs to be shared equally between recruited people, but the current conferences have the problem of too many qualified people leaving the reviewer pool after going into industry jobs. At the same time, we always have a lot of junior people who do not have the incentives to go through reviewer training. The following ideas were discussed to try to counter this trend and to have more equitable review loads:

• $\mathrm{■}$

  As an incentive, first-time reviewers could have a reduced load to help them spend more time on individual papers.

• $\mathrm{■}$

  First-time reviewers could have dedicated recognition/awards, and priority in bidding/assignments.

• $\mathrm{■}$

  First-time reviewers could be offered mentoring [63], and given the option to nominate their own PI as the mentor.

• $\mathrm{■}$

  Mentoring option in reviewer invitations: usually the invitations only have accept/decline options. There could be an option to nominate someone you would mentor.

• $\mathrm{■}$

  Venues could mandate a review load for the authors of papers submitted to a venue, with the possibility for authors who are not qualified or are contributing in other ways to be excused. For each author on submission, a form could be provided to indicate their availability as reviewers, with some common pre-set options to excuse some authors (e.g. “too junior”, “on leave”, “collaborator from a different field”, etc.).

• $\mathrm{■}$

  For papers with more than ten authors, require more authors to review.

For the deanonymization issue, the group concluded with the recommendation for the new ACL peer review committee (see subsection 4.1) to start tracking the information about preprints and intended preprints at ARR, and to monitor them over time to see what effect the new policy has, as compared to the impact of preprints reported at ACL’23 [54].

For the issue with reviewer diversity, ARR has already made an effort to broaden reviewer pool via recruitment of the authors of papers published at ACL. The effect on reviewer pool diversity needs to be estimated, and further measures for recruiting reviewers from China need to be taken if necessary. Equity of workload will be addressed at ARR by introducing a compulsory review load for authors of submitted papers.

The assignment is usually formalized as a discrete optimization problem, given the considerations of maximum load, the data about the quality of the match between submission and candidate reviewers, constraints such as conflicts of interests, reviewer seniority, experience etc., and sometimes also prevention of strategic or dishonest behavior [20, 66, 6, 62, 28, 26, 10]. The discussion focused on the experience of AAAI [33] and ACL [54], as well as the theory conferences organized on HotCRP platform [29]. It became clear that in practice each platform relied on a unique set of complex constraints, and tuning these algorithms takes a lot of effort and expertise – and the result is difficult to evaluate.

The AAAI approach [33] was particularly complex and required a lot of variables to be set by hand. The group discussed the possibility of trying to learn the optimal parameters for such a system, but there is not enough data for this.

Determining what constraints should be used for the assignments is a crucial step. The most salient component is the affinity score between (also called ‘similarity score’) the candidate reviewer and the submission, which is typically computed on the basis of reviewer publication history [6, 76, 7, 44, 41]. A major challenge is that these techniques may fail to pick up the aspects of similarity that are actually relevant for the match (e.g., abstracts can be stylistically similar but dissimilar in research topics), and NLP techniques have much room for improvement here [65]. Accordingly, the reviewer, author and chair trust in such scores is currently low [67]. Assignments based on past research may also no longer be interesting for the reviewers.

Another issue is the lack of clarity about the goal of review. Should the reviewers be optimized so that they would be most qualified to evaluate the technical correctness/soundness of the submission, or its novelty, or clarity of writing, or reproducibility, or excitement/interest to the community? It is not necessarily the case that these criteria coincide and would produce the same best match. Still, conference reviewers are implicitly expected to perform all these different roles, even though they may not be equally qualified for all this. In journals editors can craft per-paper committees; how can we do that on scale in conferences?

Another major challenge in paper-reviewer matching is noise in reviewer data (e.g., due to name disambiguation issues, unmaintained scholar profiles).

Bidding is a process commonly used to directly elicit reviewer preferences. While that allows the reviewers to pick the papers they would be the most interested in, it has integrity risks (see subsection 4.6), and is quite laborious, which is why usually people only bid on a few papers shown at the top of the list [5, 13, 40]. Furthermore, nobody would like to bid on papers that look badly written or overall umpromising, but they still need to be reviewed. Finally, people often bid on what they want to learn about, not necessarily what they have expertise on.

There is potential for improving the bidding process by imputing bids: showing the reviewers not the full set of submissions, but an imputed subset to bid on. This could be done based on affinity scores, and so as to remove various potential conflicts of interest. (A similar approach was taken in [77] for the problem of collusion rings; see [24] for an evaluation and some pros and cons of it.) To help with deanonymization, the reviewers could be asked if they follow social media a lot, and if they do – they could be only shown non-preprinted submissions.

A big problem for research on paper-reviewer matching is that each conference operates on its own format for all the data that is used as constraints in the optimization problem. Any candidate solution needs to be integrated into this specific system, and once that is done – there is no “ground truth”, so it is hard to tell which alternative is actually better [56].

It would stimulate research in this area if there was at least a unified interface for paper-reviewer matching algorithms, used by all major conference platforms. Then any new solution could be tested more easily.

In HotCRP conferences as well as more recently other conferences in AI [33], it is common to have a variable number of reviews, due to multiple rounds of review, or other reasons. There is an initial pass, with a smaller number of reviews, meant to quickly reject obvious rejects, or accept papers that are good with high confidence. Then the remaining papers are assigned more reviewers, and their goal is to consider what was not covered by the first two reviewers. Perhaps some automated analysis of initial reviews could be used to facilitate picking the new reviewers, and explaining to them why they were assigned.

• $\mathrm{■}$

  Standard interface: for the research on paper-reviewer matching to gain more traction in the community and become a research problem rather than just a conference organization problem, we need to develop a standard interface for interacting with confidential paper-reviewer matching data, that would be supported by the major conference platforms (OpenReview, Softconf, HotCRP).

• $\mathrm{■}$

  Imputing bids: The integrity, quality, and overall user experience of the bidding process can be enhanced by assigning a specific set of papers for each reviewer to evaluate and input their bidding information, and imputing the rest from it.

Participants in this working group first discussed the various current incentives for reviewing:

• $\mathrm{■}$

  Prestige service roles, e.g., being area chairs or senior area chairs.

• $\mathrm{■}$

  People can mention it on their CVs.

• $\mathrm{■}$

  Building an informal social credit with colleagues that will lead to invitations (e.g., seminars) and positive response to future service requests.

• $\mathrm{■}$

  Listing and acknowledging of reviewers in the proceedings.

• $\mathrm{■}$

  Reviewer awards.

• $\mathrm{■}$

  Conference policies forcing eligible authors to also sign up to review.

• $\mathrm{■}$

  Additional motivations include keeping up with the literature, gatekeeping or influencing directions of their field, and improving scientific report quality [42].

Participants also discussed reasons why reviewers may not want to review [42]:

• $\mathrm{■}$

  Time may be better spent elsewhere.

• $\mathrm{■}$

  Too stressful.

• $\mathrm{■}$

  Assigned papers are not interesting.

• $\mathrm{■}$

  No recognition for the work.

Based on these observations, number of potential directions towards addressing this problem were then proposed and discussed:

• $\mathrm{■}$

  Assign reviewers in a manner that area chairs know the reviewers professionally, so that there is more accountability from reviewers.

• $\mathrm{■}$

  Collect (and possibly make public) reviewer performance over time.

• $\mathrm{■}$

  Overcome challenges in measuring review quality [19].

• $\mathrm{■}$

  If measuring review quality is hard, at least incentivize other measurable desiderata like completing the reviews in time or signing up for reviewing.

• $\mathrm{■}$

  Nudge people appropriately, e.g., via personalized reminders with actual names and the link to the paper they agreed to review, or personalized thank you emails for their service.

• $\mathrm{■}$

  Develop incentive structures where people who provide good reviewers will have to review less.

As for the incentives for authors, a key challenge discussed is the high prestige often attributed solely to the act of publishing a paper. Additionally, various organizations and governments provide monetary and other forms of incentives for publishing an increased number of papers. It has been observed that “introduction of incentives by a country is associated with an increase in submissions by the country; the relation is particularly strong between cash bonuses and submissions” [14]. Therefore, there are both implicit and explicit pressures on authors to publish more frequently.

In response, some academic conferences have implemented measures to discourage excessive submissions. For example, the International Conference on Learning Representations (ICLR) publicly posts all submitted papers, including those that are rejected, along with their peer reviews. This transparency is intended to deter submissions of lower quality, as the public availability of reviews can dissuade authors from submitting subpar work. Furthermore, several conferences now require authors to include reviews from any prior rejections when resubmitting papers, which are then passed to the new reviewers. This practice aims to prevent repeated submissions of low-quality papers, as a previous rejection could negatively influence subsequent reviews [64]. Despite these measures, the effectiveness of such policies in reducing the number of submissions or improving submission quality has not been comprehensively documented or measured. Thus, developing reliable methods to assess the causal impact of these policies on submission behaviors remains a critical open issue.

Concrete next steps comprise:

• $\mathrm{■}$

  Developing more fair and accurate ways of measuring review quality, for instance, overcoming the problems discovered by the experiments in [19].

• $\mathrm{■}$

  Developing economic models capturing differences between venues where the peer-review quality is perceived to be good versus venues where it is perceived to be poor.

• $\mathrm{■}$

  Designing protocols for better longitudinal compilation of reviewer performance.

There are two primary types of incentives that need thoughtful design to support the peer-review process. First, reviewing is often a voluntary task, and creating incentives that promote high-quality reviews is a challenge. While some strategies focus on increasing the volume of reviews, such as mandating that eligible authors must participate in peer reviewing, these do not necessarily guarantee quality. More targeted approaches aim to enhance review quality, like awards for outstanding reviewers such as those at the NeurIPS conference, as well as more theoretical approaches using game theory [78, 60, 69]. However, these approaches face several hurdles, including the gap between theoretical assumptions and real-world scenarios, unclear effects of these policies on reviewer motivation, and difficulties in accurately assessing the quality of reviews [19].

The second type of incentive concerns the authors. With the high numbers of submissions to conferences, ensuring thorough and high-quality peer reviews is becoming increasingly difficult. For authors, the cost of submitting papers is relatively low: acceptance means inclusion in a prestigious conference; rejection has minimal consequences. This low-risk environment coupled with noise in the process encourages the submission of papers that may even be of unsuitable quality, which might still be accepted. Additionally, some institutions and governments reward researchers for having papers accepted at these conferences, further incentivizing high submission rates and compounding the challenges in the review process. It is thus crucial to explore incentive systems that motivate authors to submit only high-quality work. Initiatives like ICLR policy of making all submissions public can deter low-quality submissions by adding repercussions for rejection. Meanwhile, platforms like TMLR accept all competent and relevant submissions, which could diminish the prestige of mere acceptance. The effectiveness of these initiatives in maintaining high standards, however, remains to be evaluated.

Natural Language Processing (NLP) has significant potential to improve the peer review process. There are various problems in peer review for which current works rely primarily on numerical data such as ratings provided by reviewers. These include challenges like miscalibration [17, 55, 73], subjectivity [43], elicitation [45, 37], and author-identity bias [68, 3]. Although these works focus on the numeric assessments, some of the main components of peer review—submissions and feedback–are text–based, making NLP a fitting tool for analysis and improvement.

Experiments in peer review also focus on quantifiable outcomes such as ratings and acceptance decisions [31, 2, 61, 63, 50, 64, 35]. Despite the numeric focus, the textual nature of the data suggests that NLP can offer substantial contributions to these areas. While some efforts have been made [39, 15, 48], much potential remains for NLP to further address these challenges.

Moreover, NLP can help tackle issues that are currently unaddressed, such as identifying unsubstantiated criticisms in reviews or pinpointing deficiencies in papers. However, developing these NLP methods must also consider the safety and fairness of their applications, and how these methods are evaluated and measured. This area of research is gaining increasing popularity [12, 36, 30, 9, 27], and all these considerations will be explored in detail in the forthcoming paper titled “What Can Natural Language Processing Do for Peer Review?”[80] emerging from this Dagstuhl Seminar.

This working group quickly converged to the understanding that:

• $\mathrm{■}$

  There is a huge opportunity for improving peer review via latest advancements in natural language processing.

• $\mathrm{■}$

  There are also as many challenges as doing it in a safe, fair, and accurate manner, as well as in evaluating the outcomes.

• $\mathrm{■}$

  It is thus important to convey this message to researchers in NLP, ML and related communities, and also make it as easy as possible to step into this research application domain.

• $\mathrm{■}$

  A suitable means of doing so is to write a position paper, accompanied with a repository containing various datasets pertaining to peer review.

With this motivation, the remainder of the sessions focused on planning, organizing, and beginning to write the position paper and compile the datasets. The paper touches upon the following topics:

• $\mathrm{■}$

  Background of the peer-review process.

• $\mathrm{■}$

  Assistance before the review process.

  • –

    Preparing the manuscripts.

    • *

      Writing assistance for authors.

    • *

      Helping authors form metadata such as keywords and TL;DRs.

    • *

      Initial screening of manuscripts for basic checks.

  • –

    Reviewer-paper matching.

    • *

      Computing similarities between reviewers and submitted papers.

    • *

      Finding conflicts of interest.

    • *

      Reducing strategic behavior.

• $\mathrm{■}$

  Assistance during the review process.

  • –

    Evaluating certain aspects of the manuscript.

  • –

    Helping write the review.

  • –

    Discussions with authors and reviewers.

• $\mathrm{■}$

  Assistance after the review process.

  • –

    Helping with the meta review.

  • –

    Final decisions.

  • –

    Camera-ready submissions.

  • –

    Post-conference analysis.

• $\mathrm{■}$

  Data, privacy, and legal aspects.

• $\mathrm{■}$

  Measurements and experimentations.

• $\mathrm{■}$

  Ethics.

The whitepaper manuscript is available at [80] and the associated repository is available at https://github.com/OAfzal/nlp-for-peer-review.

The allure of being published in prestigious conferences can sometimes encourage unethical behaviors among participants. One concerning trend that has gained attention is the formation of collusion rings [70, 34]. In these scenarios, groups of researchers manipulate the peer review system to review each other’s papers. They then provide favorable evaluations, often disregarding the true merit of the work.

In recent years, program chairs have devoted considerable time and effort to addressing the issue of collusion rings. Tackling this challenge is essential as it directly undermines the integrity and fairness of the peer review process. Alongside this, there is a growing concern over bullying and abuse of power within the academic community. Participants reported instances where senior researchers, including some area chairs at conferences, have pressured junior colleagues to engage in these unethical activities.

One approach to addressing this issue is through technical means: developing algorithmic methods designed to detect or prevent collusion rings. Much research has already been conducted in this area [26, 77, 33, 4, 33, 24, 25, 23]. Further discussions in the working group focused on several potential strategies:

• $\mathrm{■}$

  Introducing additional conditions, such as requiring more reviewer bids. While every intervention has associated costs, it is crucial to consider potential drawbacks. For instance, although the ARR system does not involve bidding, collusion could still occur. Currently, assignments are automated with the option for Area Editors (AEs) to make changes. However, this could lead to issues if an AE is compromised.

• $\mathrm{■}$

  Implementing policies to ensure that the same person does not repeatedly review another individual’s papers over multiple years. While such interventions encourages diversity, their disadvantages must also be weighed carefully in terms of reducing expertise of assigned reviewers.

• $\mathrm{■}$

  Approaching the problem as a network issue, where more distant social links might serve as indicators of conflicts of interest. Further analyzing patterns of paper submissions and reviews, identifying discrepancies between poor-quality papers and positive reviews. Although not necessarily indicative of malicious collusion, such patterns could still pose significant problems.

• $\mathrm{■}$

  Conducting automated assessments of papers and, in cases of high disagreement between reviewers, assigning an additional reviewer. This approach must be handled carefully to avoid a high rate of false positives.

• $\mathrm{■}$

  Developing models to quantify the likelihood that certain behaviors are due to chance. This requires careful evaluation to ensure accuracy and effectiveness.

A second, human-centric approach was explored to address this problem, with several strategies proposed:

• $\mathrm{■}$

  Whistleblower Support: Participants shared knowledge about instances where collusion rings were exposed through shared communications within chat groups. This highlights the need for robust mechanisms that enable whistleblowers to safely report unethical behaviors.

• $\mathrm{■}$

  Policy Development Board: Conferences should establish a board dedicated to policy creation. This board would be responsible for developing protocols and guidelines to manage reports of misconduct effectively, including those of handling whistleblower reports.

• $\mathrm{■}$

  Guidelines for Program Chairs: Since program chairs of conferences change every year, it will be useful to develop guidelines for program chairs on what sorts of prevention measures are there for collusion rings.

• $\mathrm{■}$

  Inter-Conference Data Sharing: Some preventive measures may require data from multiple conferences to detect recurring patterns of misconduct. However, this poses legal and technical challenges, such as difficulties in transferring data between different conferences or iterations due to current platform limitations. Addressing these issues will be crucial for effective prevention.

• $\mathrm{■}$

  Education and Training: Implementing educational programs to inform researchers about unethical practices and their severe consequences can serve as a preventive measure. This approach aims to cultivate a culture of integrity and transparency within the academic community.

Next steps: Collusion rings may be addressed either by developing methods to detect them, or by preventing them (during the reviewer assignment phase itself). Although there is ongoing research on these questions [26, 33, 77, 4, 24, 23, 25], finding solutions involves significant trade-offs, and this problem largely remains open [24, 23]. Effective strategies to combat these unethical practices are crucial for maintaining the integrity of the peer-review process. There are a number of subsequent steps that are underway following the Dagstuhl Seminar. The first is to make more researchers aware of this problem and encouraging them to work on technical solutions using the tools at their disposal. For instance, NLP tools have not been used so far to combat against collusion rings, and we argue for doing so in the forthcoming paper on NLP for peer review (disussed in the NLP section of this report). We are also developing guidelines for program chairs and pushing along policies for some sharing of data across conferences. We envisage further outcomes to emanate in due time based on the aforementioned directions concieved at the Dagstuhl Seminar.

A major bottleneck for developing AI tools to support the integrity of peer review is the lack of data for training and evaluation in this sensitive domain subject to privacy and data protection. Existing large scale data collections suffer from the low-response rate. Therefore, the group participants were interested in helping with the data collection.

This involved tasks such as creating an empirical protocol for data collection, achieving the ethical and legal clearance by writing a proposal for the ACL ethics committee, etc., defining what kind of data to extract, for what purposes and how to implement this technically. Further questions that were covered in the discussion were data ownership, distribution and management, allowing for retraction of consent, backward compatibility with the previous data collection workflow, cross-community data collection, incentivizing data donation and public outreach through blog posts, social media, etc.

At the time of writing, most of the discussed topics have already been addressed by the seminar participants. The result is documented in this website: https://arr-data.aclweb.org. The data collection protocol has been successfully implemented and the data collection is now ongoing within the ACL Rolling Review (ARR) platform.