By Hendrik Hüning, Lydia Mechtenberg, and Stephanie W. Wang*

In our modern digital society, online discourse strongly influences how individuals form opinions. Anonymous strangers and friends alike engage in discussions on political or societal issues in online forums, on social media platforms, or via messenger services such as WhatsApp. Interestingly, many scholars associate the recent rise in populism over the last decade with the advent of social media (e.g. [1] and [2]). 

In order to understand the potential role of online discourse and social media in the rise of populism, researchers have to analyse large amounts of these online discourses. Especially interesting for researchers studying populism is the analysis of factual argumentation versus “unfounded” claims or the use of fallacious reasoning that more frequently appear in the argumentation of populists (e.g. [3]).

It is impossible to analyse large amounts of textual data by hand through methods such as content analysis, which requires that researchers manually classify online discourse into pre-defined categories. Alternative methods from the field of Natural Language Processing (NLP), however, allow researchers to train algorithms to automatically detect sophisticated concepts. These include arguments in natural language texts. The emerging field of Argument Mining is a central cornerstone in research about Artificial Intelligence. That is because Argument Mining combines cognitive models with statistical models for automatised argument search (see for an overview of the literature at [4] and [5]).

In a recent working paper, we investigate the usefulness of Argument Mining techniques for the analysis of deliberation in online chat discussions within a survey experiment (see [6]). Our chat data come from an online survey experiment that was conducted in two waves around the Local Rent Control Initiative ballot on the 6th of November 2018 in California. On that day, Californians could vote in favour or against a proposition that expands local governments' authority to enact rent control in their communities. Participants answered a survey and half of them were randomly chosen to take part in an online discussion about rent control in randomly assigned chat-groups of five.

We explore the machine-learning task of automatically detecting chat messages that contain justification for an underlying claim. We combine machine learning with state-of-the-art NLP techniques—namely, Word2vec, GloVe, and BERT (See [7], [8] and [9])—to train algorithms to distinguish chat messages with argumentative content from those that do not have such content. Detecting sophisticated concepts such as arguments in chats is challenging because of the brevity of messages as well as unstructured and fragmented sentences. Despite the challenges, our results are promising. The methods we used work fairly well: argumentative versus non-argumentative messages can be distinguished with an accuracy of up to 84%. 

Our research has the following implications for research on populism in online discourse. First, it enables an automated analysis of large amounts of textual data that is generated in online discussions. This helps in understanding the dynamics of (online) opinion formation, persuasion, and the development of echo chambers. Second, results from this research can be a fruitful input when it comes to developing chat-bots as neutral moderators of online debates. Health institutions have recently used chat-bots to reduce misinformation in the recent COVID-19 pandemic (See [10]). In online deliberations, however, the employment of chat-bots as neutral moderators is still in its infancy. Properly designed chat-bots could improve the deliberative quality of online discourse by highlighting arguments of chat participants, asking participants to react to others’ arguments, pointing to fallacious reasoning, or ensuring that discussions do not veer off topic.

Summing up, our research supports efforts to detect argumentation in online discourse with the help of automated methods. Argument Mining levels up research on populism because it can detect when and how opinions are supported by arguments. Moreover, it can serve as a foundation for persuasion research, i.e. it can shed light on the question of when and how arguments are persuasive, which is important in the discussion about how opinion formation in online discourses shapes our society.

*Hendrik Hüning is a postdoctoral research fellow at the Department of Economics, University of Hamburg, from where he received his PhD in 2018. His research examines the effects of communication on economic behavior using Text Mining and NLP techniques. His publications have appeared in the North American Journal of Economics and Finance and the Journal of Economic Interaction and Coordination.

*Lydia Mechtenberg’s research focuses on political economics. She received a PhD in Philosophy from the Johannes-Gutenberg University in Mainz and a PhD in Economics from the Technical University of Berlin. Since 2012, she is Professor of Economics at the University of Hamburg. Her publications have appeared in the Review of Economic Studies and Management Science, among others.

*Stephanie W. Wang is a behavioral and experimental economist who received a PhD in Economics from Princeton University. She is currently an Associate Professor of Economics at the University of Pittsburgh. She has published in American Economic Review, Econometrica, and PNAS, among others. 

Endnotes

[1] Postill, J. (2018). Populism and social media: a global perspective, Media, Culture & Society 40 (5), 754–765.

[2] Engesser, S., Ernst, N., Esser, F., Büchel, F. (2017). Populism and social media: how politicians spread a fragmented ideology, Information, Communication & Society 20 (8), 1109-1126.

[3] Blassnig, S., Büchel, F., Ernst, N., Engesser, S. (2019). Populism and Informal Fallacies: An Analysis of Right-Wing Populist Rhetoric in Election Campaigns. Argumentation 33, 107-136.

[4] Lippi, M., Torroni, P. (2016). Argumentation Mining, State of the Art and Emerging Trends, ACM Transactions on Internet Technology, 16(2), 10:1-10:25.

[5] Cabrio, E., Villata, S. (2018). Five Years of Argument Mining: a Data-driven Analysis, Proceedings of the 27^th International Joint Conference on Artificial Intelligence, July 2018, 5427-5433.

[6] Hüning, H., Mechtenberg, L., Wang, S. (2021). Detecting argumentative discourse in online chat experiments, Working Paper. Download the publication here.

[7] Mikolov, T., Chen, K., Corrado, G., Dean, J. (2013). Efficient estimation of word representations in vector space, Computer Science, 1301.3781.

[8] Pennington, J., Socher, R., Manning, C.D. (2014). Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1532–1543, Doha, Qatar.

[9] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. Computer Science, 1810.04805.

[10] Miner, A.S., Laranjo, L. & Kocaballi, A.B. (2020). Chatbots in the fight against the COVID-19 pandemic, NPJ Digital Medicine 3 (65).