Efficient and Equitable Natural Language Processing in the Age of Deep Learning (Dagstuhl Seminar 22232)

Abstract

This report documents the program and the outcomes of Dagstuhl Seminar 22232 "Efficient and Equitable Natural Language Processing in the Age of Deep Learning". Since 2012, the field of artificial intelligence (AI) has reported remarkable progress on a broad range of capabilities including object recognition, game playing, speech recognition, and machine translation. Much of this progress has been achieved by increasingly large and computationally intensive deep learning models: training costs for state-of-the-art deep learning models have increased 300,000 times between 2012 and 2018 [1]. Perhaps the epitome of this trend is the subfield of natural language processing (NLP) that over the past three years has experienced even sharper growth in model size and corresponding computational requirements in the word embedding approaches (e.g. ELMo, BERT, openGPT-2, Megatron-LM, T5, and GPT-3, one of the largest models ever trained with 175B dense parameters) that are now the basic building blocks of nearly all NLP models. Recent studies indicate that this trend is both environmentally unfriendly and prohibitively expensive, raising barriers to participation in NLP research [2,3]. The goal of this seminar was to mitigate these concerns and promote equity of access in NLP. 

References.
[1] D. Amodei and D. Hernandez. 2018. AI and Compute. https://openai.com/blog/ai-and-compute 
[2] R. Schwartz, D. Dodge, N. A. Smith, and O. Etzioni. 2020. Green AI. Communications of the ACM (CACM) 
[3] E. Strubell, A. Ganesh, and A. McCallum. 2019. Energy and Policy Considerations for Deep Learning in NLP. In Proc. of ACL.