Towards a Unified Model of Scholarly Argumentation

The basic goal of scholarly argumentation is to add knowledge to the existing knowledge of a domain/field.⁷⁷7In addition to this primary purpose, we recognize that papers are also written self-expression, career reasons, or other reasons. The way this knowledge is added (the kind of scholarly argumentation) follows the specific rules and traditions of the field. The definition of these specific rules and traditions is what we refer to as an argumentation type.

We attempt to identify different scholarly argumentation types, organized based on the domain (i.e., area of expertise) and genre (i.e., document type) of the scientific publication. Figure 2 illustrates the idea of prevalent argumentation types across different domains and different document types.

If we can acquire conceptual and empirical knowledge about the distribution illustrated in Figure 2, we will identify usage patterns across domains, and at least partially, decouple topics, domains, and argumentation types. This orthogonality can justifiably be seen as the identification of argumentation strategies.

Starting from an anecdotal study with the review of two different publications, we discuss the main gaps when annotating argumentative sentences in scientific papers.

Argumentation in scientific articles may be modeled at different levels of granularity, from the macro-level discourse structure of an entire article (e.g., in terms of elements such as model, experiments, and discussion) to the micro-level argumentative structures of individual clauses, sentences, and paragraphs. As an initial basic study, all members of the breakout group annotated the sentence-level structure of the introductions of two scientific articles from different disciplines, identifying which sentences comprise the claims and the premises of the authors’ arguments. Here, we considered a claim to be an assertion that the authors aim to sell as new, true, or similar, and a premise as a reason supporting either the claim directly or another premise.

In particular, we considered one paper each from two domains reflecting two different types of papers, namely, a corpus paper from computational linguistic [1] and an experiment paper from medical chemistry [2]. For each paper, we first annotated its introduction individually, and then we compared and discussed the results. Here, we report only on some noteworthy findings that we made.

First, we observed that these two texts share a similar argumentative agenda and structure, despite the wide divergence in subject matter and lexical content. As shown in Table 2, this rough similarity can be exposed by comparing key sentences drawn from different locations in a paper’s introductory section. These sentences exhibit intents – which we characterized in terms of putatively different audiences – that vary with their position in the discourse and are shared across the two target domains. Titles must succinctly summarize a paper’s content and, depending on the domain, may include features intended to draw interest from potential reviewers. Lead sentences typically state the overriding concern an entire field addresses, often in a language digestible by a general audience. Sentences expressing claims vary in their specificity and concreteness, ranging from concrete contribution to central insight.

As shown in the table, we distinguished between main claims (claims that the paper’s author presumably deemed most important) and proximal claims (pro forma claims that provide useful context). We found that the ability to distinguish these two types of claims relies substantially on domain expertise. For both domains, we observed that the introduction contains only very few real claims in the sense of assertions the authors aim to convince the reader of – about one to three depending on the annotator.

Initially, there was notable disagreement in the group, none of whom has extensive chemistry expertise, about which statements in the medical chemistry paper constituted claims and which of these was the main claim. In contrast, the group’s annotations of the computational linguistics paper showed considerable agreement. The only exception was the annotations of a group member with less background in computational linguistics. This member chose as the main claim a sentence that all other members viewed as proximal.

This result clearly established the importance of domain expertise for certain types of argumentative analysis. In particular, determining which is the main claim requires the reader to assess the scientific significance of a statement, an assessment that may require extensive knowledge of an area of research. However, based on our interdisciplinary discussion, we reached an agreement in most cases, even in our analysis of the chemistry paper, suggesting that the automated modeling of the argumentative structure of scientific articles is feasible, in principle. The key question is how much domain knowledge the analysis of scientific argumentation in a given discipline requires.

Our anecdotal study and discussion suggest that having domain experts for all paper types and domains may be a costly and inefficient process. Instead, we identified a new research challenge: how to accelerate the interaction with domain experts to speed up cross-domain argumentation annotation? This research challenge spans new research questions such as:

• $\mathrm{■}$

  Can we probe experts with specific portions of text instead of having them read the whole publication?

• $\mathrm{■}$

  Can we identify a specific vocabulary and use it in customized domain-specific annotation platforms?

• $\mathrm{■}$

  Can we identify a set of questions for domain experts to help guide other users in the annotation process? Examples of these questions include identifying the section where the main claim is, which are the main sections to look at first when analyzing a paper in a particular domain, what are the main types of evidence in a publication or typical lexical cues to identify claims or evidence in a given domain.

Following our anecdotal study, we explored some of these questions with our two papers, as shown in Tables 3. We believe these are initial examples that should be expanded in order to identify a wider range of commonalities in scientific literature.

Our working group discussed the Introduction section of two conference papers from computational linguistics and medical chemistry as examples to explore the discrepancy in argumentation between domains. Through manual annotation and discussion, we came to find that scientific argumentation varies among different domains noticeably. Further work may extend this analytical and comparative paradigm on scholarly argumentation to other domains, genres, and parts in the publication. After discussing the results of our anecdotal study, we believe that more research is needed to accelerate domain expert interaction for annotating argumentative sentences in different domains. Instead of asking domain experts to directly help with various task-specific works required by non-experts, their contribution would be more efficient and influential by helping summarize the universal features of argumentation for one specific genre, domain, and part. For example, experts could annotate a feasible scale of representative papers, extract lexical hints, etc.

Key questions in this area include how to structure the interaction with the domain expert for lightweight knowledge elicitation, and how to abstract, represent, and inject the features that encapsulate the knowledge required for accurate models of a given domain’s argumentation. Meanwhile, without sacrificing models’ performance, the minimum degree of domain knowledge elicitation from experts is also worth studying.

This working group focused on the evaluation of argumentation quality. We wanted to take an alternative approach to typical text synthesis evaluation. We also wanted to develop an evaluation framework that is general enough that can be applied to the numerous argumentation schemes that exist.

We settled on an approach that would determine the argumentation quality of a text through the interrogation by a Question Answering (QA) system about the argumentation within. For each genre of text, one would need to develop a series of diagnostic (and increasingly specific) questions that would reflect the quality of the reasoning or argumentation of the paper. Each genre of text would have a different set of characteristics. For example, the process for systematic reviews would ask questions about the comprehensiveness of the review, while for technical or experimental papers, questions would be related to the description of the model, and how or why it brings about an improvement in the domain.

The envisioned evaluation system would take a text to be evaluated and a question bank organised by the different genres. Depending on the genre the appropriate questions would be applied to the text and their answers rated. Note that the framework is flexible such that answers could be rated by humans and later automatically. Depending on the effectiveness of the QA system and the nature of the answers, a combination of human and automatic answer rating can be used.

We developed several initial question banks for academic papers. In Table 4 we provide a sample of what we believe to be the kinds of questions that should be asked. However, how would one judge the answers? Ideally questions would be simple and easy to judge automatically. In reality the argument is nuanced and complex. Therefore, assessing the quality of answers is likely to be a human task, though as answers become more formalised (even simplified to just yes or no questions) automated assessment becomes more feasible.

We note that it seems natural that the assessor might want to record caveats, concerns, or other thoughts. We allow the assessor to include such comments as motivation for why they assign the score they do.

To guide the development of questions, we also devised a taxonomy of question types. Table 5 contains our initial taxonomy of question types. However, in addition to question types, it is also necessary to define how answer to questions will be evaluated.

Thus finally, we devised a hierarchy of evaluation. Each level corresponds to a different interrogation method for probing argumentation quality.

1. 1.

   First level of evaluation: model evaluation as retrieval

   • $\mathrm{■}$

     Input: Open-ended questions

   • $\mathrm{■}$

     Output: “retrieval unit” i.e., sentence/snippet/etc.

2. 2.

   Second level of evaluation: model evaluation as a checklist

   • $\mathrm{■}$

     Input: Yes/No questions

   • $\mathrm{■}$

     Output: Yes/No

3. 3.

   Third level of evaluation: multiple-choice QA

   • $\mathrm{■}$

     Input: multiple-choice questions

   • $\mathrm{■}$

     Output: selection of one answer from set of answers

We have already begun the development of a tool for the community to perform offline evaluation of argumentation quality. We are developing the tool as an open source project, and is available at https://github.com/hscells/arg_eval. We plan to continue to develop this tool to support the various question types and levels of evaluation. Once we have laid the groundwork with a proper evaluation tool and expanded upon the framework proposed here, the next logical step the the development of a QA system. The first version of the QA system will focus on a small subset of the possible question types and perhaps only one level of evaluation. This will demonstrate the viability of a QA system to evaluation argumentation quality and will set a clear direction for further expansion of the QA system.

In science, peer reviewing is the deliberation process where members of a scientific community with diverse levels of experience decide if a scholarly work provides a valuable, scientific contribution [1, 2]. In the process, the actors of the community (i.e. authors, reviewers, meta-reviewers, and possibly others, e.g., chairs) exchange arguments about the strengths and weaknesses of a particular scientific contribution within multiple, direct and indirect dialogues (review, rebuttal, decision-making).

Usually, the decision-making process begins with the reviewers writing their reviews and optionally the authors responding to the reviews (i.e. rebuttal). Here, the meta-reviewer has to arrive at a decision about the promotion of acceptance of the paper. This process is mainly about weighing the arguments raised by the reviewers and happens under time constraints. To provide more efficient and effective access to (a) the content of the paper, and (b) the many arguments raised by the individual reviewers, we envision an intelligent dialogue system which answers questions of the meta-reviewer.

From an NLP perspective, this is more challenging than other domain-specific task-oriented dialog system scenarios [9], as the meta-reviewer’s needs underpinning these questions can vary from information retrieval and exploration (e.g. “What datasets did the authors use?”) to combining information from multiple sources (e.g. “According to the reviews, what are the main weaknesses of the paper?”) and summarization tasks (e.g. “Please briefly summarize the paper?”).

The goal of this breakout group was to evaluate the feasibility of collecting (training) data for such a system and to refine the task definition along the way. Having such a dataset could provide an interesting basis for studying both various facets of argumentation, like quality and convincingness of reviews or implicit ranking of the value system employed by the meta-reviewer. Furthermore, different dialogue strategies can be analyzed, like the way of gathering information in order to come to a decision.

The breakout group defined the goal of the sessions as formulating the decision-making process for a scholarly paper as a dialogue and conducted a first annotation round using an Oxford-style inspired debate format. Two groups (debaters and judges) were involved in the decision-making process. Given a paper and its reviews, the debaters discussed the pros and cons of the paper and a decision was formed by the judges. To study the relation between arguments extracted from the reviews or the underlying paper, all turns required explicit grounding in the respective documents. For instance, an argument in favor of acceptance should be substantiated by the review passage (As reviewer 1 says …) from which it was derived.

After reviewing the annotation process, it became clear that the task needs to be better aligned with the actual review process of the respective research discipline (in our example: Natural Language Processing) and in such a way that the data collected will be useful for a real-world system. There was a consensus that the debate format is obstructive as it forces dialogue partners to defend a position which might be different from their own. Additionally, the coarse granularity of groundings in natural dialogue – i.e. referring to the entire document instead of sentences or paragraphs – limited the study of the relations between argumentative units in the reviews and papers.

We revised the system’s purpose as a decision-making support system for the meta-reviewer after reviews (and rebuttals) are collected. Therefore, the dialogue involves two parties (meta-reviewer, intelligent support system) with the meta-reviewer questioning the system to inform their final decision, and the system as an oracle with knowledge of the paper, the reviews and optionally other related work. To resemble a real-world situation, a time limit is imposed on the meta-reviewer which enforces limited exposure to the reviews and paper. It is important to note that such a system will be most beneficial for papers where the decision is difficult (i.e. so-called borderline papers). Further, the system will support in weighing the reviews as there exist different levels of reviewing expertise.

Finally, we conducted another round of data collection by pairing the senior members of the group (meta-reviewers) with the junior members (imitating the dialogue system). The junior members prepared themselves by reading the papers and reviews in detail. Before the dialogue, meta-reviewers had five minutes to study the reviews. At the end of the dialogue, the meta-reviewer had to make a statement about the acceptance or rejection of the paper. We collected 16 dialogues (in English) about 4 papers involving 4 meta-reviewers and 4 system agents. The conversations were transcribed using the OpenAI Whisper [10] model which is known to have good transcription quality. However, manual post-processing was necessary as the model output is not separated based on the speaker.

In summary, we formalized the idea of a decision-support system for meta-reviewers during peer reviewing as a dialogue system. We designed and evaluated a protocol to collect dialogue data for such a system. As a result we created a dataset of 16 high-quality question & answer dialogues between a meta-reviewer and a system agent which is knowledgeable about the paper and reviews.

There exist several open questions about the future course of this project. State-of-the-art NLP models require a certain amount of data for training. However, as the data collection procedure requires the participation of expert-level reviewers (i.e. meta-reviewers), it cannot be scaled easily. One way could be to align the discussion format between reviewers and meta-reviewer during peer-reviewing with the dialogue format proposed in this group. Similar to the first pilot annotation, the study of grounding of the assistants’ turns in the review texts is one important future step towards modeling such alignments. This step might be facilitated by adding structured annotations to the reviews, indicating, for example, the targets of the comments (comments regarding specific parts of the paper, the experimental settings, etc.) or the severity of the issues raised by the reviewers [3].

Another question is whether the data collection procedure can be generalized to different use-cases. Here, the first step should be to separate task-specific components (e.g. meta-reviewer role) from the more general aspects. Further, an additional layer of annotation would help specify general and domain-specific dialogue acts. The usability of different dialogue argumentation schemata [4] needs to be assessed. Annotation can be conducted using off-the-shelf tools, like INCEPTION [5] or PEER⁸⁸8The PEER tool is an annotation tool designed for the domain of scholarly articles permitting span annotations and commenting directly inside an article’s PDF. This tool is under development at the UKP Lab, Technical University of Darmstadt; please refer to https://intertext.ukp-lab.de/ for updates on its release.

A crucial issue is the evaluation of the success of the conversation. As stated above, the goal of the system is to inform the meta-reviewer’s final decision. This is rather difficult to quantify and can be biased by other influencing factors, e.g. low-quality reviews. Conducting user studies is a possible direction but it is costly and time-consuming. Another approach could be to assess whether individual questions have been answered satisfactorily by the system rather than directly evaluating the overall conversation. While we successfully transcribed the audio data collected in this group, we recommend data collection via text-based input methods [6] to overcome the need for post-hoc manual speaker identification and enabling data collection in an online setup. Also, we point out that neither the rebuttals nor the official meta-reviews are inclduded in the peer review dataset [7] due to the complicated licensing situation with peer-reviewing data [8], but they would be another useful resource.