Document Open Access Logo

From Prediction to Precision: Leveraging LLMs for Equitable and Data-Driven Writing Placement in Developmental Education

Authors Miguel Da Corte , Jorge Baptista

PDF HTML 5

Files

Thumbnail PDF
PDF
OASIcs.SLATE.2025.1.pdf
  • Filesize: 0.66 MB
  • 18 pages
HTML5 Logo HTML (experimental)
OASIcs.SLATE.2025.1.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Social and professional topics → Adult education
  • Social and professional topics → Student assessment
  • Computing methodologies → Information extraction
  • Computing methodologies → Lexical semantics
  • Computing methodologies → Language resources
  • Computing methodologies → Natural language generation
Keywords
  • Large Language Models (LLMs)
  • Developmental Education (DevEd)
  • writing assessment
  • text classification
  • English writing proficiency

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Accurate text classification and placement remain challenges in U.S. higher education, with traditional automated systems like Accuplacer functioning as "black-box" models with limited assessment transparency. This study evaluates Large Language Models (LLMs) as complementary placement tools by comparing their classification performance against a human-rated gold standard and Accuplacer. A 450-essay corpus was classified using Claude, Gemini, GPT-3.5-turbo, and GPT-4o across four prompting strategies: Zero-shot, Few-shot, Enhanced, and Enhanced+ (definitions with examples). Two classification approaches were tested: (i) a 1-step, 3 class classification task, distinguishing DevEd Level 1, DevEd Level 2, and College-level texts in one single run; and (ii) a 2-step classification task, first separating College vs. Non-College texts before further classifying Non-College texts into DevEd sublevels. The results show that structured prompt refinement improves the precision of LLMs' classification, with Claude Enhanced + achieving 62.22% precision (1 step) and Gemini Enhanced + reaching 69.33% (2 step), both surpassing Accuplacer (58.22%). Gemini and Claude also demonstrated strong correlation with human ratings, with Claude achieving the highest Pearson scores (ρ = 0.75; 1-step, ρ = 0.73; 2-step) vs. Accuplacer (ρ = 0.67). While LLMs show promise for DevEd placement, their precision remains a work in progress, highlighting the need for further refinement and safeguards to ensure ethical and equitable placement.

Cite As Get BibTex

Miguel Da Corte and Jorge Baptista. From Prediction to Precision: Leveraging LLMs for Equitable and Data-Driven Writing Placement in Developmental Education. In 14th Symposium on Languages, Applications and Technologies (SLATE 2025). Open Access Series in Informatics (OASIcs), Volume 135, pp. 1:1-1:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.SLATE.2025.1

Author Details

Miguel Da Corte
  • University of Algarve, Campus de Gambelas, Faro, Portugal
  • INESC-ID Lisboa, Portugal
Jorge Baptista
  • University of Algarve, Campus de Gambelas, Faro, Portugal
  • INESC-ID Lisboa, Portugal

Funding

This work was supported by Portuguese national funds through FCT (Reference: UIDB/50021/2020, DOI: 10.54499/UIDB/50021/2020) and by the European Commission (Project: iRead4Skills, Grant number: 1010094837, Topic: HORIZON-CL2-2022-TRANSFORMATIONS-01-07, DOI: 10.3030/101094837).

Supplementary Materials

References

  1. Tufan Adiguzel, Mehmet Haldun Kaya, and Faith Kürşat Cansu. Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 15(3):ep429, 2023. URL: https://doi.org/10.30935/cedtech/13152.
  2. AI Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku. Claude-3 Model Card, 1:1, 2024. Google Scholar
  3. Nathan Atox and Mason Clark. Evaluating Large Language Models through the lens of linguistic proficiency and world knowledge: A comparative study. Authorea Preprints, 2024. Google Scholar
  4. Jaehyeon Bae, Seoryeong Kwon, and Seunghwan Myeong. Enhancing software code vulnerability detection using gpt-4o and claude-3.5 sonnet: A study on prompt engineering techniques. Electronics, 13(13), 2024. URL: https://doi.org/10.3390/electronics13132657.
  5. Benjamin Samuel Bloom. Taxonomy of educational objectives: Handbook II. David McKay, 1956. Google Scholar
  6. Loredana Caruccio, Stefano Cirillo, Giuseppe Polese, Giandomenico Solimando, Shanmugam Sundaramurthy, and Genoveffa Tortora. Claude 2.0 Large Language Model: Tackling a real-world classification problem with a new iterative prompt engineering approach. Intelligent Systems with Applications, 21:200336, 2024. URL: https://doi.org/10.1016/J.ISWA.2024.200336.
  7. Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. KnowPrompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference 2022, WWW '22, pages 2778-2788, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3485447.3511998.
  8. Jacob Cohen. Statistical power analysis. Hillsdale, NJ: Erlbaum, 1988. Google Scholar
  9. Miguel Da Corte and Jorge Baptista. Enhancing writing proficiency classification in Developmental Education: the quest for accuracy. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 6134-6143, 2024. URL: https://aclanthology.org/2024.lrec-main.542.
  10. Miguel Da Corte and Jorge Baptista. Leveraging NLP and machine learning for English (l1) writing assessment in Developmental Education. In Proceedings of the 16th International Conference on Computer Supported Education (CSEDU 2024), 2-4 May, 2024, Angers, France, volume 2, pages 128-140, 2024. URL: https://doi.org/10.5220/0012740500003693.
  11. Miguel Da Corte and Jorge Baptista. LLM-based writing classification prompts for 1-step and 2-step classification experiments. Technical report, University of Algarve & INESC-ID Lisboa, 2025. Google Scholar
  12. Miguel Da Corte and Jorge Baptista. Refining English writing proficiency assessment and placement in Developmental Education using NLP tools and Machine Learning. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU, pages 288-303. INSTICC, SciTePress, 2025. URL: https://doi.org/10.5220/0013351500003932.
  13. Miguel Da Corte and Jorge Baptista. Results for LLM-based 2-step classification. Technical report, University of Algarve & INESC-ID Lisboa, 2025. Google Scholar
  14. Miguel Da Corte and Jorge Baptista. Toward consistency in writing proficiency assessment: Mitigating classification variability in Developmental Education. In Proceedings of the 17th International Conference on Computer Supported Education - Volume 2: CSEDU, pages 139-150. INSTICC, SciTePress, 2025. URL: https://doi.org/10.5220/0013353900003932.
  15. Jane Denison-Furness, Stacey Lee Donohue, Annemarie Hamlin, and Tony Russell. Welcome/Not Welcome: From Discouragement to Empowerment in the Writing Placement Process at Central Oregon Community College. In Jassica Nastal, Mya Poe, and Christie Toth, editors, Writing Placement in Two-Year Colleges: The Pursuit of Equity in Postsecondary Education, pages 107-127. The WAC Clearinghouse/University Press of Colorado, 2022. URL: https://doi.org/10.37514/PRA-B.2022.1565.2.04.
  16. Nikki Edgecombe and Michael Weiss. Promoting equity in Developmental Education reform: A conversation with Nikki Edgecombe and Michael Weiss. Center for the Analysis of Postsecondary Readiness, page 1, 2024. Google Scholar
  17. Jessica López Espejel, El Hassane Ettifouri, Mahaman Sanoussi Yahaya Alassan, El Mehdi Chouham, and Walid Dahhane. GPT-3.5, GPT-4, or BARD? evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts. Natural Language Processing Journal, 5:100032, 2023. URL: https://doi.org/10.1016/J.NLP.2023.100032.
  18. John Fields, Kevin Chovanec, and Praveen Madiraju. A survey of text classification with transformers: How wide? how large? how long? how accurate? how expensive? how safe? IEEE Access, 12:6518-6531, 2024. URL: https://doi.org/10.1109/ACCESS.2024.3349952.
  19. Deen Freelon. Recal OIR: ordinal, interval, and ratio intercoder reliability as a web service. International Journal of Internet Science, 8(1):10-16, 2013. Google Scholar
  20. Mohammed Osman Gani, Ramesh Kumar Ayyasamy, Saadat M Alhashmi, Khondaker Sajid Alam, Anbuselvan Sangodiah, Khondaker Khaleduzzman, and Chinnasamy Ponnusamy. Towards enhanced assessment question classification: a study using Machine Learning, deep learning, and Generative AI. Connection Science, 37(1):2445249, 2025. URL: https://doi.org/10.1080/09540091.2024.2445249.
  21. Margherita Grandini, Enrico Bagli, and Giorgio Visani. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756, 2020. URL: https://arxiv.org/abs/2008.05756.
  22. Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with Large Language Models. In International Conference on Artificial Intelligence and Statistics, pages 5549-5581. PMLR, 2023. URL: https://proceedings.mlr.press/v206/hegselmann23a.html.
  23. Sarah Hughes and Ruth Li. Affordances and limitations of the ACCUPLACER automated writing placement tool. Assessing Writing, 41:72-75, 2019. Google Scholar
  24. Muhammad Imran and Norah Almusharraf. Google Gemini as a next generation AI educational tool: a review of emerging educational technology. Smart Learning Environments, 11(1):22, 2024. URL: https://doi.org/10.1186/S40561-024-00310-Z.
  25. Elizabeth Kopko, Jessica Brathwaite, and Julia Raufman. The next phase of placement reform: Moving toward equity-centered practice. research brief. Center for the Analysis of Postsecondary Readiness, 2022. URL: https://files.eric.ed.gov/fulltext/ED626145.pdf.
  26. Milan Kostic, Hans Friedrich Witschel, Knut Hinkelmann, and Maja Spahic-Bogdanovic. LLMs in automated essay evaluation: A case study. In Proceedings of the AAAI Symposium Series, volume 3, pages 143-147, 2024. URL: https://doi.org/10.1609/AAAISS.V3I1.31193.
  27. Gyeong-Geon Lee, Ehsan Latif, Lehong Shi, and Xiaoming Zhai. Gemini Pro defeated by GPT-4v: Evidence from education. arXiv preprint arXiv:2401.08660, 2023. Google Scholar
  28. Stephanie Link and Svetlana Koltovskaia. Automated Scoring of Writing, pages 333-345. Springer International Publishing, Cham, 2023. URL: https://doi.org/10.1007/978-3-031-36033-6_21.
  29. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in Natural Language Processing. ACM Computing Surveys, 55(9):1-35, 2023. URL: https://doi.org/10.1145/3560815.
  30. Ross Markle. Redesigning course placement in service of guided pathways. Educational Considerations, 50(2):8, 2025. Google Scholar
  31. Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in Natural Language processing via Large pre-trained Language Models: A survey. ACM Computing Surveys, 56(2):1-40, 2023. URL: https://doi.org/10.1145/3605943.
  32. Yida Mu, Ben P. Wu, William Thorne, Ambrose Robinson, Nikolaos Aletras, Carolina Scarton, Kalina Bontcheva, and Xingyi Song. Navigating prompt complexity for zero-shot classification: A study of large language models in computational social science. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12074-12086, Torino, Italia, May 2024. ELRA and ICCL. URL: https://aclanthology.org/2024.lrec-main.1055/.
  33. Sasha Nikolic, Carolyn Sandison, Rezwanul Haque, Scott Daniel, Sarah Grundy, Marina Belkina, Sarah Lyden, Ghulam M Hassan, and Peter Neal. ChatGPT, Copilot, Gemini, SciSpace and Wolfram versus higher education assessments: an updated multi-institutional study of the academic integrity impacts of Generative Artificial Intelligence (GenAI) on assessment, teaching and learning in engineering. Australasian Journal of Engineering Education, 29(2):126-153, 2024. Google Scholar
  34. Austin Pack, Alex Barrett, and Juan Escalante. Large language models and automated essay scoring of english language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6:100234, 2024. URL: https://doi.org/10.1016/J.CAEAI.2024.100234.
  35. Dolores Perin, Julia Raufman, and Hoori Santikian Kalamkarian. Developmental reading and English assessment in a researcher-practitioner partnership. Technical report, CCRC, Teachers College, Columbia University, 2015. Google Scholar
  36. Dylan Phelps, Thomas Pickard, Maggie Mi, Edward Gow-Smith, and Aline Villavicencio. Sign of the times: Evaluating the use of large language models for idiomaticity detection. In Archna Bhatia, Gosse Bouma, A. Seza Doğruöz, Kilian Evang, Marcos Garcia, Voula Giouli, Lifeng Han, Joakim Nivre, and Alexandre Rademaker, editors, Proceedings of the Joint Workshop on Multiword Expressions and Universal Dependencies (MWE-UD) @ LREC-COLING 2024, pages 178-187, Torino, Italia, May 2024. ELRA and ICCL. URL: https://aclanthology.org/2024.mwe-1.22/.
  37. Sundai Pichai and Demis Hassabis. Introducing Gemini: our largest and most capable AI model, 2023. Accessed: February 19, 2025. URL: https://blog.google/technology/ai/google-gemini-ai/#introducing-gemini.
  38. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Google Scholar
  39. Areeg Fahad Rasheed, M Zarkoosh, Safa F Abbas, and Sana Sabah Al-Azzawi. TaskComplexity: A dataset for task complexity classification with in-context learning, FLAN-T5 and GPT-4o benchmarks. arXiv preprint arXiv:2409.20189, 2024. URL: https://doi.org/10.48550/arXiv.2409.20189.
  40. Eugénio Ribeiro, Nuno Mamede, and Jorge Baptista. Exploring the automated scoring of narrative essays in brazilian portuguese using transformer models. In Proceedings of the 16th International Conference on Computational Processing of Portuguese-Vol. 2, pages 14-17, 2024. URL: https://aclanthology.org/2024.propor-2.4/.
  41. Siti Bealinda Qinthara Rony, Tan Xin Fei, and Sasa Arsovski. Educational justice. reliability and consistency of Large Language Models for automated essay scoring and its implications. Journal of Applied Learning and Teaching, 8(1), 2025. Google Scholar
  42. Kathrin Seßler, Maurice Fürstenberg, Babette Bühler, and Enkelejda Kasneci. Can ai grade your essays? a comparative analysis of large language models and teacher ratings in multidimensional essay scoring. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 462-472, 2025. URL: https://doi.org/10.1145/3706468.3706527.
  43. Arthur Spirling. Why open-source Generative AI models are an ethical way forward for science. Nature, 616(7957):413-413, 2023. Google Scholar
  44. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. Google Scholar
  45. LearnLM Team, Abhinit Modi, Aditya Srikanth Veerubhotla, Aliya Rysbek, Andrea Huber, Brett Wiltshire, Brian Veprek, Daniel Gillick, Daniel Kasenberg, Derek Ahmed, et al. Learnlm: Improving Gemini for learning. arXiv preprint arXiv:2412.16429, 2024. Google Scholar
  46. Yuqi Wang, Wei Wang, Qi Chen, Kaizhu Huang, Anh Nguyen, and Suparna De. Zero-shot text classification with knowledge resources under label-fully-unseen setting. Neurocomputing, 610:128580, 2024. URL: https://doi.org/10.1016/j.neucom.2024.128580.
  47. Zhiqiang Wang, Yiran Pang, Yanbin Lin, and Xingquan Zhu. Adaptable and reliable text classification using Large Language Models. arXiv preprint arXiv:2405.10523, 2024. Google Scholar
  48. Changrong Xiao, Wenxing Ma, Qingping Song, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Qi Fu. Human-AI collaborative essay scoring: A dual-process framework with LLMs. In Proceedings of the 15th International Learning Analytics and Knowledge Conference, pages 293-305, 2025. URL: https://doi.org/10.1145/3706468.3706507.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact