Technologies to Support Critical Thinking in an Age of Misinformation

The Internet is full of opinion hidden under tons of irrelevant and unstructured data. From micro-blogging platforms like Twitter to video repositories such as YouTube the users express sentiments about products, brands, institutions and governments. Moreover, users tag other users’ opinions: they submit comments in the same micro-blog, link them with other media content, submit brief comments to the videos or just click the “like” option. In the past few years, there has been a huge increment of interest in the analysis of this type of content by opinion consumers, such as companies and media organizations. Among others, companies aim at mining this collective opinion in order to know what people think and how they feel about products and services. The main drawback of current solutions is that they only consider the textual content, ignoring other sources and modalities of opinion and its cross-media relationships.

This talk addresses the challenge of opinion mining of multimedia content from the Web. This comprises a multi-modal analysis of social media streams and their underlying network dynamics considering different media channels such as Twitter, YouTube, Flickr, Google, and Wikipedia. It specifically proposes solutions for:

1. 1.

   The detection of trending topics from a large set of dynamic data streams. These trending topics were able to be clustered, tracked and aggregated over social media channel and over time. In particular the combination of statistical methods with linked open data was of help to achieve this goal (see [3, 4, 6]).

2. 2.

   The content-based multimedia analysis on single modalities and combination of multi-modalities. Here, the early shift from traditional approaches towards deep learning proofed to be in particular successful. Key element of analysis were Adjective Noun Pairs (ANP) providing a mid-level representation for visual content and foundation for sentiment analysis. The idea of ANPs was further extended towards Verb-Noun-Pairs to capture temporal concepts present in audio and video streams (see [1, 2, 5, 7])

Ethics, Goals, and Societal impact have always been central subjects since the early days of the field of research on artificial intelligence such as affective computing. But currently, the new uses of social robots, affective conversational agents (chatbots), and, more generally, the so-called “affectively intelligent” digital environments in fields as diverse as health, education, insurance, transport, or economics reflect a phase of significant change in human-machine relations, amplify the necessity to keep great attention in ethical dimensions of these systems. What ethical issues arise from the development of affective computing with chatbot/robot interaction? Does it raise the crucial issue of trust? How will humans co-learn, co-create and co-adapt with the Machine? Notably, how will vulnerable people be protected against potential threats of the machine? During an interaction, we adapt our linguistic behaviors but also our prosodic and gestural behaviors and our conversational strategies. This multi-level adaptation can have several functions: reinforcing engagement in interaction, emphasizing our relationship with others, and showing empathy. Anthropomorphism introduces many challenges, among them ethical, uncanny valley, practical implementation, and user mind-reading problems. The anthropomorphic goal of “just like a human-to-human conversation”. The designers of conversational agents seek for many to imitate, simulate the dialogical behavior of humans, and users spontaneously anthropomorphize the conversational agents” capacities and lend them human understanding. Thus, the Dilemma of the researchers is, on the one hand, to achieve the highest performance with conversational virtual agents and robots (close to or even exceeding human capabilities) but on the other hand, to demystify these systems by showing that they are “only machines”.

Conversational agents and social robots using autonomous learning systems and affective computing will change the game around ethics. We need to build long-term experimentation to survey Human-Machine Co-evolution and to build “ethics by design” chatbots and robots. In the chair HUMAAINE (head: L. Devillers, LISN-CNRS, France), we aim to study the Human-Machine Affective interactions and relationships, in order to audit and measure the potential influence of intelligent and affective systems on humans, and finally to go towards a conception of “ethical systems”, by design or not and to propose evaluation measures. For this purpose, the planned scientific work focuses on the detection of social emotions in a human voice, and on the study of audio and spoken language “nudges” [1, 2], intended to induce changes in the behavior of the human interlocutor.

Nudging is an ethically highly problematic topic. A digital nudge is an almost imperceptible incentive in the design of a digital system to drive behavior that is supposed to improve personal or collective well-being. Digital nudges use personal data and biometric sensors to profile and encourage people to take unintended actions while using familiar online technologies such as email, pop-ups, SMS, web interfaces, smart watches, mobile apps, IoT, home appliances, smart cars, chatbots, robots, etc. However, when a digital nudge is enhanced by Artificial Intelligence systems (so-called AI-enhanced nudge) using machine learning and affective computing technologies based on cognitive biases and behavioral science, its potential is immense. While its usage can be beneficial for an individual or the society, the AI-enhanced nudge persuasive power and intrusive capacity can also cause subliminal manipulations and profound and long-lasting changes in the behavior of users, especially children, and vulnerable people. If AI-enhanced Nudges are (intentionally or not) misused, they may become dangerous and raise serious ethical issues that can undermine the level of distrust in AI-enhanced nudging systems. AI-enhanced Nudging is already a reality influencing the actions and behaviors of thousands of people in the fields of education, health, gaming, gambling, hospitality, smart cities, security, justice, etc. However, this “soft” manipulation of behavior and emotions raises ethical questions to which no standard today provide direct answers. As AI-enhanced Nudging systems are flourishing in the market, it is widely believed that their design and the use of them, in the short or long term, ought to establish human and social responsibility through auditable behaviors under a typical set of conditions. This could help to build trustworthiness within a sustainable market. There is an urgent need to create a shared terminology and consensual processes and methodologies to mitigate and ethically adjust the enormous ability of people’s manipulation provided by AI technologies such as affective computing to digital nudge [3].

With advancements in sensing and processing power and more sophisticated machine-learning methods, computing systems can increasingly detect and monitor human activities. Computers that consider the context in which they are used can support their users according to their current location, activities, and intent, a field coined context-aware computing [9]. In our work, we have extended this notion to also include the user’s cognitive context to build systems that help users increase their ability to effectively process information according to their current mental state. By utilising phone sensor data, for example, we have trained machine-learning algorithms to detect when people are attentive to their phones [3] and seek stimulation [8]. Insights into when a person is bored or focused can provide us with a better understanding of when people are more productive and when downtimes occur: during highly focused states, devices in the user’s environment can be advised to prevent interruptions in order to help people focus better. Beyond in-situ assessments of cognitive states, we have developed tools and methods to elicit users’ circadian rhythms of alertness [1, 2, 10], which describe systematic cognitive performance fluctuations throughout the day. Awareness of these rhythms opens up a whole range of opportunities to suggest content, schedule a day full of work, or generally recommend activities whose cognitive requirements match the user’s current state [5]. The resulting tools and algorithms give insights into the user’s internal body clock, which helps people to better schedule, for example, learning sessions or generally activities that require high focus. In recent years, however, it has become apparent that effectively dealing with information is not necessarily a matter of consuming more in less time but of the quality of the information processing itself. Society is transitioning into an era where computing pervades all aspects of people’s lives, with humans and their cognitive processes at the centre of it. The rise of fake news and the interplay between bad actors, fast dissemination through social media, and people’s receptivity to emotionally charged content present an ever-growing challenge to individuals, society, and our democratic institutions [7]. When looking at what can be done about its reception, we have identified several preventative interventions to bolster people against fake news. These include media literacy training, psychological inoculation, and transaction cost economics [6]. Further, receptivity to fake news often comes from the prevalence of cognitive biases. They play an important role in how information is perceived and processed, a fact that can be both utilised and exploited by computing systems. A prominent example of a cognitive bias is the confirmation bias, i.e., the tendency to seek out information that confirms our existing perspectives and notions. We have recently established a strand of research to use sensors to detect the occurrence of cognitive biases. Computing systems capable of detecting cognitive biases, which we call bias-aware systems, can help people increase their awareness of and mitigate their effects as well as inform recommender systems to introduce a more balanced news diet. One of the main challenges is the collection of ground truth, i.e., ensuring we can successfully induce and measure the occurrence of cognitive biases for observational and experimental research. Therefore, we developed a tool to collect ground truth on people’s implicit preferences that can be adapted to any thematic issue, such as opinions on climate change, feminism, or political ideologies [4]. The Dagstuhl Seminar on “Technologies to Support Critical Thinking in an Age of Misinformation” was born out of the realisation that the problem of fake news can only be addressed in a truly interdisciplinary fashion as it involves the technology through which fake news spread, the human who creates, receives and shares it, and the regulatory bodies who are looking for ways of reeling in its spread. Throughout the 3-day seminar, our team sat down with behavioural psychologists, government advisers, and technologists to discuss the human element in this triangle of technology, human, and government. The goal of our bias detection research is to allow people to increase their awareness of their innate biases and allow systems to help mitigate them. Media literacy training, on the other hand, can help current and future generations of technology users critically process online information. Future computing systems need to be designed responsibly to consider people’s cognitive biases and help them bolster against their cognitive vulnerabilities. Technology is thus the ouroboros of fake news, i.e., its enabler and mitigator.

Democracy is in retreat or under pressure worldwide. Even in countries with strong democracies, polarization is increasing, and the public sphere is awash in misinformation and conspiracy theories. Many commentators have blamed social media and the lack of platform governance for these unfortunate trends, whereas others have celebrated the Internet as a tool for liberation, with each opinion being buttressed by supporting evidence. One way to resolve this paradox is by identifying some of the pressure points that arise between the architecture of human cognition and the online information landscape, and their fallout for the well-being of democracy. Two such pressure points arise from the algorithmic curation of content and the prevalence of misinformation and disinformation on social medial. Virtually everything users see on the internet is curated by intelligent algorithms (e.g., the newsfeed on Facebook or Twitter). These algorithms are designed by platforms without public accountability or auditing, with the primary intent of keeping users engaged longer by satisfying their presumed preferences. While preference satisfaction by itself is not a threat to democracy, it can become problematic if extremist or conspiratorial content keep a person engaged longer, because the platforms are then incentivized to prevent more and more potential harmful content. Algorithms may thus at least indirectly imperil our democracy when people are being radicalized or are presented with misinformation not because they want to, but because platforms are making money by facilitating it. Algorithmic content curation can additionally be problematic if people’s personal data are used to identify sensitive attributes, such as their personality or sexual orientation, which political operatives can then exploit by presenting messages to people that exploit their personal vulnerabilities. This process is known as microtargeting and it comes with a number of attributes that may imperil democracy. In the absence of any regulation, one possible countermeasure involves “boosting” people’s ability to detect on their own when they might be targeted by manipulative messages. A existence proof of boosting showed that once people were given information about their own personality along the introversion-extraversion spectrum, they were better able to identify advertisements that were aimed at them based on their personality. Similar “boosting” approaches can also equip people to become resilient to misinformation and disinformation online. This approach is known as inoculation and it entails warning people ahead of time that they might be misled, and providing them with information about the misleading rhetorical techniques they are likely to encounter. Inoculation has been shown to be effective in numerous different domains, from anti-vaccination messages to radicalization attempts and conspiracy theories. In all cases, people’s ability to detect when they are manipulated was significantly enhanced by inoculation. Notwithstanding the success of such cognitive countermeasures, they are insufficient to counter the immense asymmetry in power between platforms and users that currently exists and that gives rise to the pressure points between cognition and technology. It requires deep structural change and smart regulation to create a new Internet with democratic credentials.

Different countries are taking very different approaches to regulation of social media to combat misinformation. Singapore has passed a law against telling lies which allows the government to order the take down of material regarded as untrue. China is seeking to register anyone who comments publicly about key political issues online. The European Union is bringing in regulation that makes social media responsible for harms which include harms to democracy and civic discourse as well as harms to fundamental rights. The UK is proposing more limited regulation that focuses on immediate harm to individuals rather than harm to society (but which still might capture medical misinformation). The US is adopting a more laissez-faire attitude based on giving primacy of freedom of speech. However, within the US, individual platforms are implementing their own governance mechanisms in recognition of public pressure for change.

These regulatory strategies do not specifically call for action on critical thinking. However, platforms may respond to the European regulatory proposals by adopting measures such as misinformation vaccination. (Also, there are, in some territories, complementary strategies on media literacy alongside regulatory proposals – the EU strategy on disinformation).

Key limitations to the successful implementation of regulation are: A lack of social consensus around the meanings of the words used (e.g. “harm to public discourse” “psychological harm”). Lack of agreed mechanisms that are capable of determining whether such harm has occurred. Lack of mechanisms for determining responsibility. (Regulations require platforms to balance rights to free speech against risk of harm. The issue for regulators is whether they have found the right balance. This requires a determination of whether or not they could have done better in balancing these risks which, in turn, requires an understanding of what is possible in order to make a sound assessment of responsibility.)

Each of these issues is exacerbated by the rapidly changing and hugely heterogenous nature of the harms being addressed, as well as the complexity of the environment that is being regulated.

Regulators will likely have to adopt an approach based on identifying the most egregious harms and using rough and ready measures to assess the responsibility of the platform. The degree to which this will significantly impact disinformation and online harms is uncertain.

Regulators would be wise to adopt a strong stance at the outset with regard to the data access provisions in the EU regulations. They should set out a long term strategy to establish

• $\mathrm{■}$

  relatively objective/consensual approaches to categorising and monitoring misinformation;

• $\mathrm{■}$

  research methods to understand the impact of misinformation on individuals and on democracy;

• $\mathrm{■}$

  and mechanisms for understanding the relative impact of different types of remedy including media literacy and critical thinking.

The efforts toward technologies to support critical thinking in an age of misinformation require a collaborative effort across the fields of Technology, Society, and Policy. In this appetiser talk, I will outline some of the primary challenges faced in each area, pointing to promising research that indicates opportunities for moving forward.

TECHNOLOGY
Challenges faced within the technology field include biases, biased algorithms, and black box decision-making. Therefore, it is critical to recognise the real-world consequences of algorithmic systems. Examples include disparities in AI skin cancer diagnoses between different skin colours and discrimination built into the design of a fraud detection system of the Dutch tax authorities. Recent work by Huszár et al. analyses the amplification of tweets by elected legislators from major political parties in seven countries [1]. Their results show that the mainstream political right enjoys higher algorithmic amplification. While highlighting the possibility of assessing the impact of algorithmic-driven recommendation systems, it also raises new questions, including; Should distribution always be a perfect 50/50 split? Are politics as black and white as left / right? What can we do about this technological bias?

SOCIETY
Disinformation, filter bubbles, and an increased polarisation in politics and beyond are amongst the challenges currently faced in society. Current events, such as the Russian invasion of Ukraine, bring to the front the societal challenges related to disinformation. Disinformation also plays a significant role domestically, with the US being a famous example of the growing ideological divide between Democrats and Republicans. In democratic countries with a multi-party democratic system, such as The Netherlands, polarisation can take a different form – with traditional parties finding it increasingly challenging to distinguish themself from one another. Recent work by Broockman and Kalla studies the effect of paying Fox News viewers to regularly watch CNN (two politically opposed media channels) [2]. Compared to a control group of Fox News viewers, the study finds more nuanced political beliefs and knowledge of current events. The authors highlight how the skewing of media has had a broader and negative impact on how US society functions. Highlighting the opportunity for viewers to obtain more nuanced viewpoints once presented with an alternative media source, the study raises new relevant questions, including the potential for long-term effects and the impact of removing the financial incentives offered in the study.

POLICY
With the growing impact of technology in an increasingly unstable world, policy is often looked at as the instrument to bring back some stability. Simultaneously, policy can be perceived as a slow-moving instrument which struggles in dealing with local versus global issues. In this context, we increasingly see the global impact of national and international governmental organisations. In particular, the US and the EU are at the forefront of developing AI policies and research plans. A recent review by Jobin et al. studies the global landscape of AI ethics guidelines [3]. Their results highlight eleven unique ethical principles that are discussed within these guidelines, including “transparency”, “justice”, and “privacy”. Their study shows an apparent interest of both governmental organisations and industry in developing ethics guidelines while also presenting questions for further research. For example; How do we implement these guidelines in products and services? Can we match policy with outcomes? How do these, primarily developed in Europe and North America, impact the rest of the world?

Most of the research regarding misinformation that is spread online typically focuses on news and fake news (generally of a political or societal nature). While this is certainly an important and interesting topic, my focus is in knowledge-sharing in technical domains. One of the most fruitful aspects of the Seminar was the wide-ranging discussions held about trust, authority, and misinformation in academic research. In an age in which numerous papers use opaque data mining techniques and questionable data science, it is difficult to assign veracity to each research article. Doing so requires both technical (the academic field may vary) and domain expertise (the area on which the data is extracted may require specialized information) from a reader to call into question what is likely true and what is possibly not. The group discussion was very interesting and participants explored their experiences with problematic research areas and claims. It seems that what we think we know is often a product of repetition. If venues publish and promote research that have questionable conclusions based on flawed methodology, this can perpetuate more research using the same flawed techniques. This eventually creates something that we think as objective truth in the field.

This has spurred an interest in investigating the network relationship between published papers we can identify as having questionable methodologies and conclusions and their impact on the wider research community. This hopefully will uncover how deeply misinformation spreads in academic research and allow us to develop critical thinking tools for academics.