Transparent Quantitative Research as a User Interface Problem

Many scientific fields face a replication crisis: A sizable portion of quantitative research studies could not be replicated. When these studies were re-run with higher statistical power (i.e., more participants), their results yielded effects substantially weaker or even opposite of that in the original studies. This lack of replicability threatens the credibility of research claims and undermines the general public’s trust in science. The replication crisis motivated the Open Science movement that promotes transparency throughout the scientific process: research funding, research design, data collection and analysis, peer reviewing, and knowledge dissemination. These phenomena attracted the interest of researchers in the fields of Human–Computer Interaction (HCI) and Visualization (VIS) for two reasons. Like other fields, HCI and VIS researchers face challenges in promoting transparency among their peers, effectively implementing and educating transparent practices, and incorporating transparency in the research evaluation processes. However, HCI and VIS researchers have the methods and skills to empirically study these phenomena and design potential solutions. The fields of HCI and VIS also provide a challenging testbed for these inventions.

This Dagstuhl seminar initiated and advanced works on these issues by bringing together 23 researchers from HCI, VIS, statistics, psychology, data science, and philosophy. They were from Australia, Austria, Canada, Denmark, Finland, France, Germany, the Netherlands, Sweden, Switzerland, the UK, and the USA. Three participants joined online due to the COVID situation and travel difficulties.

We worked in groups to identify problem areas and prototype potential solutions in a Hackathon. We solicited feedback on these prototypes from conference and journal editors and community leaders. The seminar unfolded as follows:

Day 1:

After a brief introduction to the purpose of the seminar and the overall plan, participants discussed in small groups to identify problems and challenges to work on in the Hackathon. These discussions were intentionally designed to be free-form to avoid prematurely limiting the areas of interest. To stimulate discussions and spark ideas, we provided the participants access to free-text responses to a survey on the perception of research transparency that we collected from HCI researchers in the weeks before the seminar. Four rounds of discussion were interleaved with three-minute presentations of intermediate results in the plenary to facilitate convergence and consolidation.

In each plenary round, we also asked a few participants to interview each other in front of the room to acquaint everyone with their background and research interest. Day 1 concluded with four clusters of topics to be worked on: (1) Educating researchers, (2) Clarifying the threats from the lack of transparency, (3) Clarifying the “transparency” concept, and (4) Working on how to influence policy and procedures in the publication process.

Day 2:

Participants joined the problem cluster according to their interests and started the Hackathon. We provided each group with collaborative workspaces on Google Docs and Miro (an online whiteboard platform). After two Hackathon sessions in the morning, we further stimulated their work with an input lecture from Tim Errington, the Senior Director of Research at the Center for Open Science (see below for an abstract). This lecture highlighted challenges in promoting research transparency and provided a framework for changing research culture at multiple levels: from top-down research funding policy and bottom-up to ease the implementation of transparent practices by providing infrastructure and incentives. After the lecture, the Hackathon continued. We wrapped up the day with a 3-minute presentation from each group and a plenary discussion.

Day 3:

The Hackathon continued in the morning. We gave the participants prompts to encourage them to hone in on a concrete idea and realize a prototype that demonstrates the idea’s essence. The afternoon is free time for the participants to self-organize group activities to promote trust and informal interactions. We did not organize an excursion because the transportation companies were unavailable.

Day 4:

The Hackathon continued in the morning. In the afternoon, the participants presented their preliminary results to four panelists who joined online. The panelists hold influential positions in the research publication process in HCI and VIS: the SIGCHI President, the TOCHI editor-in-chief, the TVCG Associate Editor in Chief and Eurographics Publication Board, CG&A Associate Editor-in-chief, TVCG Associate Editor, and the vice-chair of the IEEE VIS Steering Committee. A discussion on feedback from policy-making perspectives followed each presentation. The conversation with the panelists broadened participants’ views about stakeholders and potential concerns. After the discussion, there was a plenary discussion to process the input from the panel collectively. We identified four areas to work on in the manifesto: definition, benefits, subfield-suitability, and progressive transparency.

Day 5:

Participants worked in groups to draft a manifesto on research transparency. The seminar concluded with a plenary session where we identified possible future projects, their follow-up actions, and coordinators.

The tangible results of the seminar comprise four prototypes from the Hackathon and a draft manifesto:

1. 1.

   To influence research funders’ policies, we drafted a list of policy suggestions for incentivizing research transparency and Open Science.

2. 2.

   To inspire researchers and students, we prototyped how we could collect, analyze, and showcase papers in the visualization field that are exemplary in their transparent practices.

3. 3.

   To improve infrastructure, we identified low-hanging fruits in improving the user interfaces of the ethical review and publication processes to encourage transparent practices.

4. 4.

   To ease the adoption of transparency practices, we prototyped a cheat sheet that provides reminders for considering transparent practices at each of the research stages. The cheat sheet also provides pointers to relevant guides and resources.

The draft manifesto clarifies the definition of research transparency, describes its benefit, calls for each subfield to identify its suitable set of transparent practices, and argues how transparent practices could be viewed as a progression instead of demanding everyone to be perfect at their first try. These results provide a starting point for future follow-up research and educational activities that will advance the understanding and adoption of research transparency in HCI, VIS, and beyond.

In large-scale replication studies such as in Psychology and Cancer Biology, many replication studies yielded a smaller effect size than the original study [1, 2]. Low replicability challenges the credibility of science. Replicability is associated with several research best practices, such as preregistration, using large samples, and sharing research materials [3]. However, this knowledge seems inadequate to change scientists’ behaviors widely. To change the research culture, we need to address both the lack of know-how and the lack of motivation. Both top-down and bottom-up efforts are necessary as shown in Figure 1: Funders should design incentives and policies that will change the norm of research practices. Conversely, research communities must establish infrastructures and invent technologies to facilitate these practices.

Contributors in alphabetical order:

We reviewed the existing policies from funding agencies: Australia ARC and NHMRC, European ERC, the Netherlands’ NWO, and UK’s UKRI. Although these policies encourage open-access publication and sharing of research data, they lack other transparency practices. We also found the policies focus on goals without adequately providing resources and infrastructure. Therefore, we drafted seven policy recommendations. For each recommendation, we discussed the status quo, its benefits, possible barriers, and possible improvements. Below is a summary of each recommendation:

Contributors in alphabetical order:

We saw an opportunity to encourage transparent research practices by curating a set of examples from visualization research. These examples will serve as a starting point for teaching transparent research, motivating visualization researchers to practice research transparency, and motivating the community to invent new creative transparent methods. Toward these goals, we identified forms of transparency: (1) research process and methods, (2) artifacts, (3) data, and (4) claims and limitations. Combining these forms with a taxonomy of visualization contribution types (Munzner, 2008) resulted in a matrix for the examples. As a prototype for this seminar, we brainstormed and discussed how some papers we knew fit into this matrix.

From the panelists’ feedback, we realized the importance of clarifying that these examples show the possibilities of transparency. Some contribution types may lend themselves to some forms of transparent practices easier than others. The paper should be framed as a recommendation instead of a checklist to avoid alienating some research contributions.

For the next step, we plan to collect examples that span this matrix by conducting a survey targeting visualization experts, and we drafted survey questions. We plan to publish this work in IEEE Computer Graphics and Applications (CG&A) as a viewpoint paper with extensive supplementary materials on OSF.

Contributors in alphabetical order:

“Transparency in research refers to honesty and clarity in all communications about the research processes and outcomes – to the extent possible.”

We unpack each facet of this definition as follows:

Honesty and clarity

are both necessary. A paper that is very clear but makes misleading claims is not transparent. But a paper written by authors who are fully honest but unskilled at clear communication is not fully transparent either.

Communications

include communications between the researchers and their colleagues, peers, institutions, the press, and the general public.

The research process

includes all the known, crucial decisions made to achieve the reported outcome.

The research outcomes

include research materials (including data and software), findings, and communication artifacts (research papers, videos). Being transparent about research outcomes includes sharing material (e.g., code and data) but also being clear about the limitations of the research.

To the extent possible.

Transparent research practices operate within ethical, resource, legal, and other constraints. These include ethics constraints (such as participant rights and protections), legal constraints (such as data protection laws), and resource constraints (such as access to data repositories). We also acknowledge that there is information that is not accessible by the researchers; we can only ask them to communicate the information they can reasonably know.

Contributors in alphabetical order:

Transparent research practices support the refinement, reuse, and verifiability of published research as well as its extension through follow-on research and meta-analyses. Transparent practices also support the use of research materials and findings for instruction and can increase the visibility, perceived credibility, and citation of both individual research findings and broader research areas. Transparent sharing of materials may also allow for more error detection and correction, thus fostering a larger error-correction culture. Furthermore, making research materials available to other researchers to use in their contexts will reduce duplication of effort while also helping grow and diversify research communities.

Contributors in alphabetical order:

Transparent research is not about rigid box-checking. Not all transparent research practices apply to all kinds of research, and the specific implementation of any given transparent practice depends on the subfield and research type. The practices that are useful for, for example, in-depth interview studies differ from those that are useful for iterative design & user testing studies or for large-scale online experimental work. Therefore, we do not want to make specific, one-size-fits-all recommendations. Instead, we suggest questions researchers can ask themselves and their field. Thinking about the typical research process and outputs in your field, consider the following:

1. 1.

   What elements would help build credibility for your work for reviewers and readers?

2. 2.

   If another researcher wanted to build on your work, what elements would be helpful?

These questions could be used to build transparency guidelines for a particular subfield or guide an individual researcher in implementing transparent practices in their own research.

Contributors in alphabetical order:

Becoming transparent requires our research community to give up old research practices and adopt new research methods. Transparency is a progressive process that starts with small steps before it becomes a long-term, well-established practice. It can seem intimidating at first, but here are some steps that can guide you on your journey to making your research practice more transparent: