Document Open Access Logo

Gender Bias Evaluation in English-Portuguese Automated Translation Outputs

Authors Xiaolan Xu , Sara Mendes

PDF

File

Thumbnail PDF
PDF
OASIcs.SLATE.2026.8.pdf
  • Filesize: 0.49 MB
  • 14 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Machine translation
  • Social and professional topics → Gender
Keywords
  • Gender Bias
  • Large Language Models (LLMs)
  • Neural Machine Translation (NMT)
  • English-Portuguese Translation

Metrics

Abstract

Automated translation (AT) plays a pivotal role in breaking down language barriers and fostering cross-cultural communication. However, gender bias in AT remains a pressing concern, particularly for language pairs where the source language generally does not have grammatical gender and the target language does. In this context, the present research aims to assess gender bias in English-Portuguese AT outputs. Two machine translation (MT) systems and two LLMs are evaluated in this study, examining the correlation between translated gender and societal stereotypes, and how the different models handle nouns whose gender is unknown. The outputs of two commercial MT systems (Google Translate and DeepL Translator) and two general-purpose LLMs (ChatGPT-4o and DeepSeek-V3), translating from English into European Portuguese, were analyzed. We used a subset of the WinoMT challenge set to assess gender rendering accuracy across three conditions: gender-defined nouns, gender-undefined nouns, and anaphoric pronoun resolution. Our main goal is therefore to evaluate the accuracy of gender information transmission and to understand the extent to which gender bias is present in AT outputs. Our results indicate that all four models struggle to faithfully convey gender information from source to target, with male-centric outputs predominating across conditions and MT systems showing a more pronounced bias. This work contributes a preliminary cross-system comparison for an underexplored language pair and lays the groundwork for larger-scale evaluations of gender-inclusive AT.

Cite As Get BibTex

Xiaolan Xu and Sara Mendes. Gender Bias Evaluation in English-Portuguese Automated Translation Outputs. In 15th Symposium on Languages, Applications and Technologies (SLATE 2026). Open Access Series in Informatics (OASIcs), Volume 144, pp. 8:1-8:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/OASIcs.SLATE.2026.8

Author Details

Xiaolan Xu
  • Faculty of Arts and Humanities, University of Lisbon, Portugal
Sara Mendes
  • Center of Linguistics of the University of Lisbon and Faculty of Arts and Humanities, University of Lisbon, Portugal

Funding

We acknowledge the support of Fundação para a Ciência e a Tecnologia (UID/00214: Centro de Linguística da Universidade de Lisboa).

Supplementary Materials

References

  1. Sofia Bonifácio, Helena Moniz, and Luisa Coheur. Diving into gender translation bias for the portuguese language. In 28th European Conference on Artificial Intelligence, ECAI 2025, including 14th Conference on Prestigious Applications of Intelligent Systems, PAIS 2025, pages 1067-1074. IOS Press BV, 2025. Google Scholar
  2. Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. On measuring gender bias in translation of gender-neutral pronouns. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 173-181, 2019. Google Scholar
  3. Greville G Corbett. Gender. Cambridge University Press, 1991. Google Scholar
  4. Marta R Costa-Jussà, Carlos Escolano, Christine Basta, Javier Ferrando, Roser Batlle, and Ksenia Kharitonova. Interpreting gender bias in neural machine translation: Multilingual architecture matters. In Proceedings of the aaai conference on artificial intelligence, volume 36, pages 11855-11863, 2022. URL: https://doi.org/10.1609/AAAI.V36I11.21442.
  5. Eve Fleisig and Christiane Fellbaum. Mitigating gender bias in machine translation through adversarial learning. arXiv preprint arXiv:2203.10675, 2022. URL: https://doi.org/10.48550/arXiv.2203.10675.
  6. Joel Escudé Font and Marta R Costa-Jussa. Equalizing gender bias in neural machine translation with word embeddings techniques. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 147-154, 2019. Google Scholar
  7. Charles F Hockett. A course in modern linguistics. The Macmillan Company, 1958. Google Scholar
  8. Anna Kibort and Greville G Corbett. Grammatical features inventory. Typology of grammatical features, 2008. Google Scholar
  9. Shahar Levy, Koren Lazar, and Gabriel Stanovsky. Collecting a large-scale gender bias dataset for coreference resolution and machine translation. In Findings of the association for computational linguistics: EMNLP 2021, pages 2470-2480, 2021. URL: https://doi.org/10.18653/V1/2021.FINDINGS-EMNLP.211.
  10. Maja Popović and Ekaterina Lapshinova-Koltunski. Gender and bias in amazon review translations: by humans, mt systems and chatgpt. In Proceedings of the 2nd International Workshop on Gender-Inclusive Translation Technologies, pages 22-30, 2024. Google Scholar
  11. Marcelo OR Prates, Pedro H Avelar, and Luís C Lamb. Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, 32(10):6363-6381, 2020. URL: https://doi.org/10.1007/S00521-019-04144-6.
  12. Randolph Quirk and David Crystal. A comprehensive grammar of the English language. Pearson Education India, 2010. Google Scholar
  13. Eduardo Buzaglo Raposo, Maria Fernanda Nascimento, Maria Antónia Mota, Luísa Segura, and Amália Mendes. Gramática do Português. Fundação Calouste Gulbenkian, Lisboa, 2013. Google Scholar
  14. Aleix Sant, Carlos Escolano, Audrey Mash, Francesca De Luca Fornaciari, and Maite Melero. The power of prompts: Evaluating and mitigating gender bias in mt with llms. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 94-139, 2024. Google Scholar
  15. Danielle Saunders and Bill Byrne. Reducing gender bias in neural machine translation as a domain adaptation problem. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics, pages 7724-7736, 2020. URL: https://doi.org/10.18653/V1/2020.ACL-MAIN.690.
  16. Beatrice Savoldi, Sara Papi, Matteo Negri, Ana Guerberof-Arenas, and Luisa Bentivogli. What the harm? quantifying the tangible impact of gender bias in machine translation with a human-centered study. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18048-18076, 2024. URL: https://doi.org/10.18653/V1/2024.EMNLP-MAIN.1002.
  17. Peter Siemund. Pronominal gender in English. Routledge London, 2008. Google Scholar
  18. Gabriel Stanovsky, Noah A Smith, and Luke Zettlemoyer. Evaluating gender bias in machine translation. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1679-1684, 2019. URL: https://doi.org/10.18653/V1/P19-1164.
  19. Katie Wales. Personal pronouns in present-day English. Cambridge University Press, 1996. Google Scholar
  20. Xiaolan Xu and Sara Mendes. xu-xiaolan/gender-bias-en-pt-at. Dataset, (visited on 2026-07-22). URL: https://github.com/xu-xiaolan/gender-bias-en-pt-at
    Software Heritage Logo archived version
    full metadata available at: https://doi.org/10.4230/artifacts.27322
  21. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 2979-2989, 2017. URL: https://doi.org/10.18653/V1/D17-1323.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact