Document Open Access Logo

Semi-Supervised Annotation of Portuguese Hate Speech Across Social Media Domains

Authors Raquel Bento Santos, Bernardo Cunha Matos, Paula Carvalho , Fernando Batista , Ricardo Ribeiro

PDF
Thumbnail PDF

File

OASIcs.SLATE.2022.11.pdf
  • Filesize: 0.52 MB
  • 14 pages

Document Identifiers

Author Details

Raquel Bento Santos
  • INESC-ID Lisbon, Portugal
  • Instituto Superior Técnico, Lisbon, Portugal
Bernardo Cunha Matos
  • INESC-ID Lisbon, Portugal
  • Instituto Superior Técnico, Lisbon, Portugal
Paula Carvalho
  • INESC-ID Lisbon, Portugal
Fernando Batista
  • INESC-ID Lisbon, Portugal
  • Iscte - University Institute of Lisbon, Portugal
Ricardo Ribeiro
  • INESC-ID Lisbon, Portugal
  • Iscte - University Institute of Lisbon, Portugal

Funding

This research was supported by Fundação para a Ciência e a Tecnologia, through the Projects HATE Covid-19 (Proj.759274510), MIMU (FCT PTDC/CCI-CIF/32607/2017), and UIDB/50021/2020.

Cite As Get BibTex

Raquel Bento Santos, Bernardo Cunha Matos, Paula Carvalho, Fernando Batista, and Ricardo Ribeiro. Semi-Supervised Annotation of Portuguese Hate Speech Across Social Media Domains. In 11th Symposium on Languages, Applications and Technologies (SLATE 2022). Open Access Series in Informatics (OASIcs), Volume 104, pp. 11:1-11:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2022) https://doi.org/10.4230/OASIcs.SLATE.2022.11

Abstract

With the increasing spread of hate speech (HS) on social media, it becomes urgent to develop models that can help detecting it automatically. Typically, such models require large-scale annotated corpora, which are still scarce in languages such as Portuguese. However, creating manually annotated corpora is a very expensive and time-consuming task. To address this problem, we propose an ensemble of two semi-supervised models that can be used to automatically create a corpus representative of online hate speech in Portuguese. The first model combines Generative Adversarial Networks and a BERT-based model. The second model is based on label propagation, and consists of propagating labels from existing annotated corpora to the unlabeled data, by exploring the notion of similarity. We have explored the annotations of three existing corpora (CO-HATE, ToLR-BR, and HPHS) in order to automatically annotate FIGHT, a corpus composed of geolocated tweets produced in the Portuguese territory. Through the process of selecting the best model and the corresponding setup, we have tested different pre-trained embeddings, performed experiments using different training subsets, labeled by different annotators with different perspectives, and performed several experiments with active learning. Furthermore, this work explores back translation as a mean to automatically generate additional hate speech samples. The best results were achieved by combining all the labeled datasets, obtaining 0.664 F1-score for the Hate Speech class in FIGHT.

Subject Classification

ACM Subject Classification
  • Social and professional topics → Hate speech
  • Theory of computation → Semi-supervised learning
  • Computing methodologies → Transfer learning
Keywords
  • Hate Speech
  • Semi-Supervised Learning
  • Semi-Automatic Annotation

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Supplementary Materials

References

  1. Hala Al Kuwatly, Maximilian Wich, and Georg Groh. Identifying and measuring annotator bias based on annotators' demographic characteristics. In Proceedings of the Fourth Workshop on Online Abuse and Harms, pages 184-190. Association for Computational Linguistics, November 2020. URL: https://doi.org/10.18653/v1/2020.alw-1.21.
  2. Safa Alsafari and Samira Sadaoui. Semi-Supervised Self-Training of Hate and Offensive Speech from Social Media. Applied Artificial Intelligence, pages 1-25, October 2021. URL: https://doi.org/10.1080/08839514.2021.1988443.
  3. Fabienne Baider and Maria Constantinou. Covert hate speech: A contrastive study of greek and greek cypriot online discussions with an emphasis on irony. Journal of Language Aggression and Conflict, 8(2):262-287, 2020. Google Scholar
  4. Djamila Romaissa Beddiar, Md Saroar Jahan, and Mourad Oussalah. Data expansion using back translation and paraphrasing for hate speech detection. Online Social Networks and Media, 24:100153, 2021. URL: https://doi.org/10.1016/j.osnem.2021.100153.
  5. Claudia Breazzano, Danilo Croce, and Roberto Basili. MT-GAN-BERT : Multi-Task and Generative Adversarial Learning for sustainable Language Processing. In Proceedings of the Fifth Workshop on Natural Language for Artificial Intelligence (NL4AI 2021). CEUR Workshop Proceedings, November 2021. URL: http://ceur-ws.org/Vol-3015/.
  6. Paula Carvalho, Danielle Caled, Cláudia Silva, Fernando Batista, and Ricardo Ribeiro. The expression of Hate Speech against Afro-descendant, Roma and LGBTQ+ communities in YouTube comments. Discourse and Society, submitted. Google Scholar
  7. Paula Carvalho, Bernardo Matos, Raquel Santos, Fernando Batista, and Ricardo Ribeiro. Hate Speech Dynamics Against African descent, Roma and LGBTQI Communities in Portugal. LREC, 2022. Google Scholar
  8. Danilo Croce, Giuseppe Castellucci, and Roberto Basili. GAN-BERT: Generative adversarial learning for robust text classification with a bunch of labeled examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2114-2119, Online, July 2020. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/2020.acl-main.191.
  9. Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. Proceedings of the 11th International Conference on Web and Social Media, ICWSM 2017, pages 512-515, 2017. URL: http://arxiv.org/abs/1703.04009.
  10. Ashwin Geet D'Sa, Irina Illina, Dominique Fohr, Dietrich Klakow, and Dana Ruiter. Label Propagation-Based Semi-Supervised Learning for Hate Speech Classification. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 54-59. Association for Computational Linguistics, November 2020. URL: https://doi.org/10.18653/v1/2020.insights-1.8.
  11. Tim Fitzsimons. Nearly 1 in 5 hate crimes motivated by anti-LGBTQ bias, FBI finds. NBC News, November 2019. URL: https://www.nbcnews.com/feature/nbc-out/nearly-1-5-hate-crimes-motivated-anti-lgbtq-bias-fbi-n1080891.
  12. Paula Fortuna and Sérgio Nunes. A survey on automatic detection of hate speech in text. ACM Computing Surveys, 51(4):1-30, July 2019. URL: https://doi.org/10.1145/3232676.
  13. Paula Fortuna, João Rocha da Silva, Juan Soler-Company, Leo Wanner, and Sérgio Nunes. A hierarchically-labeled Portuguese hate speech dataset. In Proceedings of the Third Workshop on Abusive Language Online, pages 94-104, Florence, Italy, August 2019. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/W19-3510.
  14. Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media, 2018. Google Scholar
  15. Akshita Jha and Radhika Mamidi. When does a compliment become sexist? analysis and classification of ambivalent sexism using twitter data. In Proceedings of the second workshop on NLP and computational social science, pages 7-16, 2017. Google Scholar
  16. György Kovács, Pedro Alonso, and Rajkumar Saini. Challenges of hate speech detection in social media. SN Computer Science, 2(2):1-15, 2021. Google Scholar
  17. Ritesh Kumar, Atul Kr Ojha, Shervin Malmasi, and Marcos Zampieri. Benchmarking aggression identification in social media. In Proceedings of the first workshop on trolling, aggression and cyberbullying (TRAC-2018), pages 1-11, 2018. Google Scholar
  18. Joao A Leite, Diego F Silva, Kalina Bontcheva, and Carolina Scarton. Toxic language detection in social media for brazilian portuguese: New dataset and multilingual analysis. arXiv preprint, October 2020. URL: http://arxiv.org/abs/2010.04543.
  19. Changchun Li, Ximing Li, and Jihong Ouyang. Semi-Supervised Text Classification with Balanced Deep Representation Distributions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5044-5053. Association for Computational Linguistics, August 2021. URL: https://doi.org/10.18653/v1/2021.acl-long.391.
  20. Martina Miliani, Giulia Giorgi, Ilir Rama, Guido Anselmi, and Gianluca E Lebani. DANKMEMES@ EVALITA 2020: The Memeing of Life: Memes, Multimodality and Politics. In EVALITA, 2020. Google Scholar
  21. Yassine Ouali, Céline Hudelot, and Myriam Tami. An Overview of Deep Semi-Supervised Learning. arXiv:2006.05278, pages 1-43, June 2020. URL: http://arxiv.org/abs/2006.05278.
  22. Maria Papadaki. Data Augmentation Techniques for Legal Text Analytics. Master’s thesis, Athens University of Economics and Business, October 2017. URL: http://nlp.cs.aueb.gr/theses.html.
  23. Haeyoun Park and Iaryna Lyshyn. L.G.B.T. people are more likely to be targets of hate crimes than any other minority group. The New York Times, June 2016. URL: https://www.nytimes.com/interactive/2016/06/16/us/hate-crimes-against-lgbt.html.
  24. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011. URL: https://scikit-learn.org/.
  25. Wyatt Ronan. New FBI Hate Crimes Report Shows Increases in Anti-LGBTQ Attacks. Human Rights Campaign, November 2020. URL: https://www.hrc.org/press-releases/new-fbi-hate-crimes-report-shows-increases-in-anti-lgbtq-attacks.
  26. Diana Santos and Alberto Simões. Portuguese-English word alignment: some experiments. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco, May 2008. European Language Resources Association (ELRA). URL: http://www.lrec-conf.org/proceedings/lrec2008/pdf/760_paper.pdf.
  27. Sheikh Muhammad Sarwar and Vanessa Murdock. Unsupervised Domain Adaptation for Hate Speech Detection Using a Data Augmentation Approach. arXiv:2107.12866, July 2021. URL: http://arxiv.org/abs/2107.12866.
  28. Connor Shorten and Taghi M. Khoshgoftaar. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data, 6(1), July 2019. URL: https://doi.org/10.1186/s40537-019-0197-0.
  29. Alexandra A. Siegel. Online hate speech. In Joshua A. Tucker Nathaniel Persily, editor, Social Media and Democracy, chapter 4, page 67. Cambridge University Press, August 2021. URL: https://doi.org/10.1017/9781108890960.
  30. Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. BERTimbau: Pretrained BERT Models for Brazilian Portuguese. In Brazilian Conference on Intelligent Systems, pages 403-417. Springer, 2020. Google Scholar
  31. Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learning, 109:373-440, November 2020. URL: https://doi.org/10.1007/s10994-019-05855-6.
  32. Zeerak Waseem. Are You a Racist or Am I Seeing Things? Annotator Influence on Hate Speech Detection on Twitter. In Proceedings of the First Workshop on NLP and Computational Social Science, pages 138-142. Association for Computational Linguistics, November 2016. URL: https://doi.org/10.18653/v1/w16-5618.
  33. Michael Wiegand, Josef Ruppenhofer, and Elisabeth Eder. Implicitly Abusive Language – What does it actually look like and why are we not getting there? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 576-587. Association for Computational Linguistics, June 2021. URL: https://doi.org/10.18653/v1/2021.naacl-main.48.
  34. Wenjie Yin and Arkaitz Zubiaga. Towards generalisable hate speech detection: a review on obstacles and solutions. PeerJ Computer Science, 7:e598, 2021. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact