Document Open Access Logo

Assessing LLMs for Culturally Sensitive Communication in European Portuguese

Authors Alice Vieira , Raquel Amaro , Lyndon Nixon

PDF

File

Thumbnail PDF
PDF
OASIcs.SLATE.2026.10.pdf
  • Filesize: 1 MB
  • 14 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Natural language processing
  • Computing methodologies → Language resources
  • Computing methodologies → Machine learning
  • Human-centered computing → Human computer interaction (HCI)
Keywords
  • LLM
  • culturally sensitive communication
  • evaluation

Metrics

Abstract

This paper investigates the ability of LLMs to identify culturally sensitive communication in European Portuguese, in comparison with human performance. The study addresses cross-cultural communication issues arising within a single language, including miscommunication, bias, framing, and stereotyping. The task is formulated as a multi-component problem combining detection, classification, span identification, and reformulation of problematic content. A dataset of 60 sentences, balanced between problematic and neutral instances, was constructed from real-world corpora and annotated by expert linguists. Three open-weight multilingual models were evaluated alongside human annotators with diverse backgrounds. Results show that LLMs achieve high accuracy in detecting problematic content, outperforming non-expert participants and approaching expert performance. However, both models and humans exhibit low agreement in the classification of communication issues, reflecting the conceptual overlap between categories. While models generate consistent reformulations, they show limitations in span precision and contextual interpretation. The findings highlight a gap between detection and interpretation capabilities and support the use of LLMs as assistive tools in cross-cultural communication tasks, particularly in politically sensitive contexts.

Cite As Get BibTex

Alice Vieira, Raquel Amaro, and Lyndon Nixon. Assessing LLMs for Culturally Sensitive Communication in European Portuguese. In 15th Symposium on Languages, Applications and Technologies (SLATE 2026). Open Access Series in Informatics (OASIcs), Volume 144, pp. 10:1-10:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/OASIcs.SLATE.2026.10

Author Details

Alice Vieira
  • CLUNL – Research Center for Linguistics at NOVA University Lisbon, NOVA University Lisbon, Portugal
Raquel Amaro
  • CLUNL – Research Center for Linguistics at NOVA University Lisbon, NOVA University Lisbon, Portugal
Lyndon Nixon
  • Storypact GmbH, Vienna, Austria

Funding

  • Vieira, Alice: European Commission (Project MultiPoD, ref. 101178821, https://multipod-project.eu/), and Portuguese national funds through FCT (CLUNL, DOI:10.54499/UID/03213/2025).
  • Amaro, Raquel: European Commission (Project MultiPoD, ref. 101178821, https://multipod-project.eu/), and Portuguese national funds through FCT (CLUNL, DOI:10.54499/UID/03213/2025).
  • Nixon, Lyndon: European Commission (Project MultiPoD, ref. 101178821, https://multipod-project.eu/).

References

  1. Muhammad Aleem, Iqra Zahoor, and Muhammad Naseem. Towards culturally adaptive large language models in mental health: Using chatgpt as a case study. In Companion of the 2024 Computer-Supported Cooperative Work and Social Computing (CSCW Companion '24). ACM, 2024. URL: https://doi.org/10.1145/3678884.3681858.
  2. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 610-623. ACM, 2021. URL: https://doi.org/10.1145/3442188.3445922.
  3. Jan Blommaert. The Sociolinguistics of Globalization. Cambridge University Press, 2010. Google Scholar
  4. Bram Bulte and Ayla Rigouts Terryn. Llms and cultural values: The impact of prompt language and explicit cultural framing. Computational Linguistics, 2026. URL: https://doi.org/10.1162/COLI.a.583.
  5. Vassiliy Cheremetiev, Quang Long Ho Ngo, Chau Ying Kot, Alina Elena Baia, and Andrea Cavallaro. Specializing general-purpose llm embeddings for implicit hate speech detection across datasets. In Proceedings of the ACM Workshop on Detection of Harmful Online Content, 2025. Google Scholar
  6. Jithin Cheriyan, Bastin Tony Roy Savarimuthu, and Stephen Cranefield. Towards offensive language detection and reduction in four software engineering communities. In Proceedings of the Evaluation and Assessment in Software Engineering (EASE 2021), pages 254-259. ACM, 2021. URL: https://doi.org/10.1145/3463274.3463805.
  7. Abdelkader El Mekki, Hiba Atou, Onur Nacar, Samer Shehata, and Muhammad Abdul-Mageed. Nilechat: Towards linguistically diverse and culturally aware llms for local communities. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), pages 10967-10991, 2025. Google Scholar
  8. Norman Fairclough. Critical Discourse Analysis: The Critical Study of Language. Routledge, 2nd edition, 2013. Google Scholar
  9. Jürgen Habermas. The Theory of Communicative Action, volume 1. Beacon Press, 1984. Google Scholar
  10. Geert Hofstede and Michael Minkov. Values survey module 2013, 2013. Accessed: 2026-05-19. URL: https://geerthofstede.com/research-and-vsm/vsm-2013/.
  11. Juliane House. Translation Quality Assessment: Past and Present. Routledge, 2015. Google Scholar
  12. Dirk Hovy and Shannon L. Spruit. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pages 591-598. ACL, 2016. URL: https://doi.org/10.18653/v1/P16-2096.
  13. Ronald Inglehart, Christian Haerpfer, Alejandro Moreno, Christian Welzel, Kseniya Kizilova, Juan Diez-Medrano, Marta Lagos, Pippa Norris, Eduard Ponarin, and Bi Puranen. World values survey: Round six - country-pooled datafile, 2014. Version available at URL: https://www.worldvaluessurvey.org/WVSDocumentationWV6.jsp.
  14. Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Knowledge and Data Engineering, 33(2):494-514, 2021. URL: https://doi.org/10.1109/TKDE.2019.2940204.
  15. Daniel Jurafsky and James H. Martin. Speech and Language Processing. Pearson, 3rd edition, 2023. Draft version available online. Google Scholar
  16. Philipp Koehn. Neural Machine Translation. Cambridge University Press, 2020. Google Scholar
  17. Stephen C. Levinson. Pragmatics. Cambridge University Press, 1983. Google Scholar
  18. Cheng Li, Ming Chen, Jiahui Wang, Sunayana Sitaram, and Xianchao Xie. Culturellm: Incorporating cultural differences into large language models. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024), 2024. Google Scholar
  19. Chen Cecilia Liu, Hiba Arnaout, Nils Kovačić, Dana Atzil-Slonim, and Iryna Gurevych. Tailored emotional llm-supporter: Enhancing cultural sensitivity. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), pages 535-574, 2026. Google Scholar
  20. Xiaofei Luo. Cross-cultural adaptation framework for enhancing large language model outputs in multilingual contexts. Journal of Advanced Computing Systems, 3(5):48-62, 2023. URL: https://doi.org/10.69987/JACS.2023.30505.
  21. Meera Narayan, Jonathan Pasmore, Eduardo Sampaio, Vignesh Raghavan, Subhajit Maity, and George Waters. Mitigating bias in large language models through culturally-relevant llms. In Proceedings of the 2025 IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS 2025), 2025. URL: https://doi.org/10.1109/ETHICS65148.2025.11098204.
  22. Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2463-2473. ACL, 2019. URL: https://doi.org/10.18653/v1/D19-1250.
  23. Timo Schick and Hinrich Schütze. It’s not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2021), pages 2339-2352. Association for Computational Linguistics, 2021. URL: https://doi.org/10.18653/v1/2021.naacl-main.185.
  24. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3407-3412. ACL, 2019. URL: https://doi.org/10.18653/v1/D19-1339.
  25. Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Ng. Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Mirella Lapata and Hwee Tou Ng, editors, Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254-263, Honolulu, Hawaii, October 2008. Association for Computational Linguistics. URL: https://aclanthology.org/D08-1027/.
  26. Helen Spencer-Oatey. Culturally Speaking: Culture, Communication and Politeness Theory. Continuum, 2nd edition, 2008. Google Scholar
  27. Helen Spencer-Oatey and Peter Franklin. Intercultural Interaction: A Multidisciplinary Approach to Intercultural Communication. Palgrave Macmillan, 2009. Google Scholar
  28. Jenny Thomas. Cross-cultural pragmatic failure. Applied Linguistics, 4(2):91-112, 1983. URL: https://doi.org/10.1093/applin/4.2.91.
  29. Teun A. van Dijk. Discourse and Power. Palgrave Macmillan, 2008. Google Scholar
  30. Max Van Doren and Claire Holland. "be my cheese?": Assessing cultural nuance in multilingual llm translations, 2025. Appen report. Google Scholar
  31. Ruth Wodak. The discourse-historical approach. In Ruth Wodak and Michael Meyer, editors, Methods of Critical Discourse Analysis, pages 63-94. Sage, 2001. Google Scholar
  32. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415, 2016. URL: https://doi.org/10.1162/tacl_a_00107.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact