Document Open Access Logo

Cross-Language Text Readability Assessment: Leveraging Multilingual Models for Improved Performance in CEFR-Level Classification

Authors Eugénio Ribeiro , Jorge Baptista

PDF

File

Thumbnail PDF
PDF
OASIcs.SLATE.2026.3.pdf
  • Filesize: 0.51 MB
  • 13 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Natural language processing
  • Applied computing → Education
Keywords
  • Readability
  • Text Complexity
  • CEFR
  • Multilinguality

Metrics

Abstract

The automatic assessment of text readability and the classification of texts by levels is essential for language education and language industries that rely on effective communication. This study explores cross-language automatic readability level classification using the levels defined by the Common European Framework of Reference for Languages (CEFR). We investigate the potential of using data in one language to improve classification performance in different languages and, thus, optimize the utilization of the limited labeled resources available for each language. We rely on a pre-trained multilingual Transformer-based language model, by fine-tuning it on annotated data in one language or in a combination of languages, and then assessing its ability to generalize even to unseen languages. In an additional scenario, we further fine-tune the models on data in the target language, to assess whether the models trained on data in different languages can capture generic information regarding text readability and then be further specialized to capture the specific characteristics of the target language. Our experiments covering the English, Dutch, and German languages revealed that direct generalization to unseen languages is challenging. However, when paired with data in the target language, multilingual data can be leveraged to capture cross-language aspects of text readability, leading to more robust and better-performing models.

Cite As Get BibTex

Eugénio Ribeiro and Jorge Baptista. Cross-Language Text Readability Assessment: Leveraging Multilingual Models for Improved Performance in CEFR-Level Classification. In 15th Symposium on Languages, Applications and Technologies (SLATE 2026). Open Access Series in Informatics (OASIcs), Volume 144, pp. 3:1-3:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/OASIcs.SLATE.2026.3

Author Details

Eugénio Ribeiro
  • INESC-ID Lisboa, Portugal
  • Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR, Lisboa, Portugal
Jorge Baptista
  • INESC-ID Lisboa, Portugal
  • Faculdade de Ciências Humanas e Sociais, Universidade do Algarve, Faro, Portugal

Funding

This work was supported by Portuguese national funds through Fundação para a Ciência e a Tecnologia, I.P. (FCT) under projects UID/50021/2025 (DOI: https://doi.org/10.54499/UID/50021/2025), UID/PRR/50021/2025 (DOI: https://doi.org/10.54499/UID/PRR/50021/2025), and UID/04466/2025 (DOI: https://doi.org/10.54499/UID/04466/2025); and by the European Commission (Project: iRead4Skills, Grant number: 1010094837, Topic: HORIZON-CL2-2022-TRANSFORMATIONS-01-07, DOI: https://doi.org/10.3030/101094837).

References

  1. Wafa Aissa, Raquel Amaro, David Antunes, Thibault Bañeras-Roux, Jorge Baptista, Alejandro Catala, Luís Correia, Thomas François, Marcos Garcia, Mario Izquierdo-Álvarez, Nuno Mamede, Vasco Martins, Miguel Neves, Eugénio Ribeiro, Sandra Rodriguez, and Elodie Vanzereven. The iRead4Skills Intelligent Complexity Analyzer. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP): System Demonstrations, pages 73-84, 2025. URL: https://doi.org/10.18653/v1/2025.emnlp-demos.6.
  2. Wafa Aissa, Thibault Bañeras-Roux, Elodie Vanzeveren, Lingyun Gao, Rodrigo Wilkens, and Thomas François. Assessing French Readability for Adults with Low Literacy: A Global and Local Perspective. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 20528-20550, 2025. URL: https://doi.org/10.18653/v1/2025.emnlp-main.1036.
  3. Soroosh Akef, Amália Mendes, Detmar Meurers, and Patrick Rebuschat. Investigating the Generalizability of Portuguese Readability Assessment Models Trained Using Linguistic Complexity Features. In Proceedings of the International Conference on Computational Processing of Portuguese (PROPOR), pages 332-341, 2024. URL: https://aclanthology.org/2024.propor-1.34/.
  4. Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton. Readability Assessment for Text Simplification. In Proceedings of the NAACL-HLT Workshop on Innovative Use of NLP for Building Educational Applications, pages 1-9, 2010. URL: https://aclanthology.org/W10-1001.
  5. António Branco, João Rodrigues, Francisco Costa, João Silva, and Rui Vaz. Rolling out Text Categorization for Language Learning Assessment Supported by Language Technology. In Proceedings of the International Conference on the Computational Processing of the Portuguese Language (PROPOR), pages 256-261, 2014. URL: https://doi.org/10.1007/978-3-319-09761-9_29.
  6. Mark Breuker. CEFR Labelling and Assessment Services. In Georg Rehm, editor, European Language Grid: A Language Technology Platform for Multilingual Europe, pages 277-282. Springer International Publishing, 2023. URL: https://doi.org/10.1007/978-3-031-17258-8_16.
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020. URL: https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  8. Miriam Cha, Youngjune Gwon, and HT Kung. Language Modeling by Clustering with Word Embeddings for Text Readability Assessment. In Proceedings of the Conference on Information and Knowledge Management (CIKM), pages 2003-2006, 2017. URL: https://doi.org/10.1145/3132847.3133104.
  9. Council of Europe. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge University Press, 2001. URL: https://rm.coe.int/1680459f97.
  10. Scott A. Crossley, Stephen Skalicky, Mihai Dascalu, Danielle S. McNamara, and Kristopher Kyle. Predicting Text Comprehension, Processing, and Familiarity in Adult Readers: New Approaches to Readability Formulas. Discourse Processes, 54(5-6):340-359, 2017. URL: https://doi.org/10.1080/0163853x.2017.1296264.
  11. Pedro Curto, Nuno Mamede, and Jorge Baptista. Automatic Text Difficulty Classifier. In Proceedings of the International Conference on Computer Supported Education (CSEDU), volume 1, pages 36-44, 2015. URL: https://doi.org/10.5220/0005428300360044.
  12. Jacob Devlin, Ming-Wei Chang, Lee Kenton, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT, volume 1, pages 4171-4186, 2019. URL: https://doi.org/10.18653/v1/N19-1423.
  13. William H. DuBay. The Principles of Readability. Impact Information, 2004. URL: https://files.eric.ed.gov/fulltext/ED490073.pdf.
  14. Anna Filighera, Tim Steuer, and Christoph Rensing. Automatic Text Difficulty Estimation Using Embeddings and Neural Networks. In Proceedings of the European Conference on Technology Enhanced Learning (EC-TEL), pages 335-348, 2019. URL: https://doi.org/10.1007/978-3-030-29736-7_25.
  15. Luciana Forti, Giuliana Grego Bolli, Filippo Santarelli, Valentino Santucci, and Stefania Spina. MALT-IT2: A New Resource to Measure Text Difficulty in Light of CEFR Levels for Italian L2 Learning. In Proceedings of Language Resources and Evaluation Conference (LREC), pages 7204-7211, 2020. URL: https://aclanthology.org/2020.lrec-1.890.
  16. Thomas François and Cédrick Fairon. An "AI readability" Formula for French as a Foreign Language. In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 466-477, 2012. URL: https://aclanthology.org/D12-1043.
  17. Thomas François, Adeline Müller, Eva Rolin, and Magali Norré. AMesure: A Web Platform to Assist the Clear Writing of Administrative Texts. In Proceedings of the Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP): System Demonstrations, pages 1-7, 2020. URL: https://doi.org/10.18653/V1/2020.AACL-DEMO.1.
  18. Gemma Team. Gemma: Open Models based on Gemini Research and Technology. Computing Research Repository, arXiv:2403.08295, 2024. URL: https://doi.org/10.48550/arXiv.2403.08295.
  19. Nicolas Hernandez, Nabil Oulbaz, and Tristan Faine. Open Corpora and Toolkit for Assessing Text Readability in French. In Proceedings of the Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI), pages 54-61, 2022. URL: https://aclanthology.org/2022.readi-1.8.
  20. Joseph Marvin Imperial, Abdullah Barayan, Regina Stodden, Rodrigo Wilkens, Ricardo Muñoz Sánchez, Lingyun Gao, Melissa Torgbi, Dawn Knight, Gail Forey, Reka R. Jablonkai, Ekaterina Kochmar, Robert Joshua Reynolds, Eugénio Ribeiro, Horacio Saggion, Elena Volodina, Sowmya Vajjala, Thomas François, Fernando Alva-Manchego, and Harish Tayyar Madabushi. UniversalCEFR: Enabling Open Multilingual Research on Language Proficiency Assessment. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9703-9755, 2025. URL: https://doi.org/10.18653/v1/2025.emnlp-main.491.
  21. Simon Jönsson, Evelina Rennes, Johan Falkenjack, and Arne Jönsson. A Component Based Approach to Measuring Text Complexity. In Proceedings of the Swedish Language Technology Conference (SLTC), pages 58-61, 2018. URL: https://sltc2018.blogs.dsv.su.se/files/2019/11/Final-compilation-of-abstracts-SLTC-2018-Nov-19.pdf.
  22. Nikolay Karpov, Julia Baranova, and Fedor Vitugin. Single-sentence Readability Prediction in Russian. In Proceedings of the International Conference on Analysis of Images, Social Networks and Texts (AIST), pages 91-100, 2014. URL: https://doi.org/10.1007/978-3-319-12580-0_9.
  23. J. Peter Kincaid, Robert P. Fishburne Jr, Richard L. Rogers, and Brad S. Chissom. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Technical report, Institute for Simulation and Training, University of Central Florida, 1975. URL: https://doi.org/10.21236/ada006655.
  24. Sidney Evaldo Leal, Magali Sanches Duran, Carolina Evaristo Scarton, Nathan Siegle Hartmann, and Sandra Maria Aluísio. NILC-Metrix: Assessing the Complexity of Written and Spoken Language in Brazilian Portuguese. Computing Research Repository, arXiv:2201.03445, 2021. URL: https://doi.org/10.48550/arXiv.2201.03445.
  25. Bruce W. Lee, Yoo Sung Jang, and Jason Lee. Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 10669-10686, 2021. URL: https://doi.org/10.18653/v1/2021.emnlp-main.834.
  26. Fengkai Liu and John SY Lee. Hybrid models for sentence readability assessment. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications (BEA), pages 448-454, 2023. URL: https://doi.org/10.18653/v1/2023.bea-1.37.
  27. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach. Computing Research Repository, arXiv:1907.11692, 2019. URL: https://doi.org/10.48550/arXiv.1907.11692.
  28. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. Supervised and Unsupervised Neural Approaches to Text Readability. Computational Linguistics, 47(1):141-179, 2021. URL: https://doi.org/10.1162/coli_a_00398.
  29. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G.C. de Souza, Alexandra Birch, and André F.T. Martins. EuroLLM: Multilingual Language Models for Europe. Procedia Computer Science, 255:53-62, 2025. URL: https://doi.org/10.1016/j.procs.2025.02.260.
  30. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed Representations of Words and Phrases and their Compositionality. In NeurIPS, pages 3111-3119, 2013. URL: https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.
  31. Salar Mohtaj, Babak Naderi, and Sebastian Möller. Overview of the GermEval 2022 Shared Task on Text Complexity Assessment of German Text. In Proceedings of the GermEval Workshop on Text Complexity Assessment of German Text, pages 1-9, 2022. URL: https://aclanthology.org/2022.germeval-1.1.
  32. Farah Nadeem and Mari Ostendorf. Estimating Linguistic Complexity for Science Texts. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, pages 45-55, 2018. URL: https://doi.org/10.18653/v1/W18-0505.
  33. Kai North, Marcos Zampieri, and Matthew Shardlow. Lexical Complexity Prediction: An Overview. ACM Computing Surveys, 55(9):1-42, 2023. URL: https://doi.org/10.1145/3557885.
  34. Ildikó Pilán and Elena Volodina. Investigating the Importance of Linguistic Complexity Features Across Different Datasets Related to Language Learning. In Proceedings of the Workshop on Linguistic Complexity and Natural Language Processing, pages 49-58, 2018. URL: https://aclanthology.org/W18-4606.
  35. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAI Blog, 1(8):9, 2019. URL: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
  36. Sandra Rodríguez Rey, André Bernárdez Braña, and Marcos Garcia. Exploring Linguistic Features in a New Readability Corpus for Spanish. Procesamiento del lenguaje natural, 74:221-239, 2025. URL: http://journal.sepln.org/sepln/ojs/ojs/index.php/pln/article/view/6677.
  37. Sandra Rodríguez Rey and Marcos Garcia. Clasificación Automática de Textos por Niveis de Lecturabilidade: Recursos e Modelos para o Galego. Linguamática, 17(2):33-56, 2025. URL: https://doi.org/10.21814/lm.17.2.488.
  38. Robert Reynolds. Insights from Russian Second Language Readability Classification: Complexity-Dependent Training Requirements, and Feature Evaluation of Multiple Categories. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, pages 289-300, 2016. URL: https://doi.org/10.18653/v1/W16-0534.
  39. Eugénio Ribeiro, David Antunes, Nuno Mamede, and Jorge Baptista. Exploring Few-Shot Approaches to Automatic Text Complexity Assessment in European Portuguese. Journal of the Brazilian Computer Society, 31(1):690-710, 2025. URL: https://doi.org/10.5753/jbcs.2025.5820.
  40. Eugénio Ribeiro, Nuno Mamede, and Jorge Baptista. Automatic Text Readability Assessment in European Portuguese. In Proceedings of the International Conference on Computational Processing of Portuguese (PROPOR), pages 97-107, 2024. URL: https://aclanthology.org/2024.propor-1.10/.
  41. Eugénio Ribeiro, Nuno Mamede, and Jorge Baptista. Avaliação Automática do Nível de Complexidade de Textos em Português Europeu. Linguamática, 16(2):121-145, 2024. URL: https://doi.org/10.21814/lm.16.2.449.
  42. Eugénio Ribeiro, Nuno Mamede, and Jorge Baptista. Text Readability Assessment in European Portuguese: A Comparison of Classification and Regression Approaches. In Proceedings of the International Conference on Computational Processing of Portuguese (PROPOR), pages 551-557, 2024. URL: https://aclanthology.org/2024.propor-1.59/.
  43. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. Computing Research Repository, arXiv:1910.01108, 2019. URL: https://doi.org/10.48550/arXiv.1910.01108.
  44. Rodrigo Santos, João Rodrigues, António Branco, and Rui Vaz. Neural Text Categorization with Transformers for Learning Portuguese as a Second Language. In Proceedings of the Portuguese Conference on Artificial Intelligence (EPIA), pages 715-726, 2021. URL: https://doi.org/10.1007/978-3-030-86230-5_56.
  45. Valentino Santucci, Filippo Santarelli, Luciana Forti, and Stefania Spina. Automatic Classification of Text Complexity. Applied Sciences, 10(20):7285, 2020. URL: https://doi.org/10.3390/app10207285.
  46. Yao Ting Sung, Wei Chun Lin, Scott Benjamin Dyson, Kuo En Chang, and Yu Chia Chen. Leveling L2 Texts through Readability: Combining Multilevel Linguistic Features with the CEFR. The Modern Language Journal, 99(2):371-391, 2015. URL: https://doi.org/10.1111/modl.12213.
  47. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. Computing Research Repository, arXiv:2302.13971, 2023. URL: https://doi.org/10.48550/arXiv.2302.13971.
  48. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), volume 30, pages 5998-6008, 2017. URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  49. Rodrigo Wilkens, David Alfter, Xiaoou Wang, Alice Pintard, Anaïs Tack, Kevin Yancey, and Thomas François. FABRA: French Aggregator-Based Readability Assessment Toolkit. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 1217-1233, 2022. URL: https://aclanthology.org/2022.lrec-1.130.
  50. Rodrigo Wilkens, Patrick Watrin, Rémi Cardon, Alice Pintard, Isabelle Gribomont, and Thomas François. Exploring Hybrid Approaches to Readability: Experiments on the Complementarity between Linguistic Features and Transformers. In Findings of the Association for Computational Linguistics: EACL, pages 2316-2331, 2024. URL: https://doi.org/10.18653/v1/2024.findings-eacl.153.
  51. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. Computing Research Repository, arXiv:1910.03771, 2019. URL: https://doi.org/10.48550/arXiv.1910.03771.
  52. Menglin Xia, Ekaterina Kochmar, and Ted Briscoe. Text Readability Assessment for Second Language Learners. In Proceedings of the Workshop on Innovative Use of NLP for Building Educational Applications, pages 12-22, 2016. URL: https://doi.org/10.18653/v1/W16-0502.
  53. Kevin Yancey, Alice Pintard, and Thomas Francois. Investigating Readability of French as a Foreign Language with Deep Learning and Cognitive and Pedagogical Features. Lingue e Linguaggio, 20(2):229-258, 2021. URL: https://doi.org/10.1418/102814.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact