Efficient and Equitable Natural Language Processing in the Age of Deep Learning

This talk presents forays into efficient QA models, modeling sparsity for hardware acceleration, and issues in measuring energy consumption.

Training modern neural networks is time-consuming, expensive, and energy-intensive. As neural network architectures double in size every few months, it is difficult for researchers and businesses without immense budgets to keep up. In this talk, I describe one approach for managing this challenge: changing the training algorithm itself. While many companies and researchers are focused on building hardware and systems to allow existing algorithms to run faster in a mathematically equivalent fashion, there is nothing sacred about this math. To the contrary, training neural networks is inherently approximate, relying on noisy data, convex optimizers in nonconvex regimes, and ad hoc tricks and hacks that seem to work well in practice for reasons that elude us.

I discuss how we have put this approach into practice at MosaicML, including the dozens of algorithmic changes we have studied (which are freely available open source), the science behind how these changes interact with each other (the composition problem), and how we evaluate whether these changes have been effective. I will also detail several surprises we have encountered and lessons we have learned along the way. In the four months since we began this work in earnest, we have reduced the training times of standard computer vision models by 7x and standard language models by 2x on publicly available cloud instances, and we believe we are just scratching the surface.

Papers about a new approximation (i.e. faster for some loss in quality) often claim the quality loss is small, while better papers perform a Pareto comparison. Unfortunately, the baseline approximations used for the Pareto comparison are usually restricted to the same type of method, such as pruning. I argue the correct baseline is all approximations that already exist. Approximations are stackable, so the question is really whether the proposed approximation belongs to a set of stacked approximations that advance the Pareto frontier. This is a high standard and difficult for the average paper to reach, so I present a partial solution. The efficient machine translation shared task establishes the state-of-the-art by soliciting competitive submissions and comparing them. Starting from a range of already efficient systems provides a much stronger baseline for evaluating a new approximation.

Our field is currently characterized by a “bigger is better” trend in the size of deep neural networks. This talk posits that this is an unsustainable recipe – akin to building a ladder to the moon. We discuss some important directions for revisiting the efficiency of our representation learning approaches.

There are many approaches to mapping natural-language sentences to symbolic meaning representations. The current dominant approach is with neural sequence-to-sequence models, which map the sentence to a string version of the meaning representation. Seq2seq models work well for many NLP tasks, including tagging and parsing, and deliver excellent accuracy on broad-coverage semantic parsing as well. However, it has recently been found that seq2seq models struggle with “compositional generalization”: They have a hard time generalizing from training examples to structurally similar unseen test sentences. I will show some new results that pinpoint this difficult more precisely, and discuss what this means for how to best evaluate semantic parsers.

I will then present our own research on compositional semantic parsing, which combines neural models with the Principle of Compositionality from theoretical semantics. Our semantic parser uses a neural supertagger to predict word meanings and a neural dependency parser to predict the compositional structure, and then evaluates this dependency structure in a graph algebra to obtain the meaning representation. We achieve state-of-the-art parsing accuracy across a number of graphbanks, at a speed of up to 10k tokens/second.

In this work, we explore rational activation functions for training transformer models. In contrast to activation functions such as GELU which remain fixed after initialization, rational activation functions are capable of approximating any arbitrary activation function during training. In preliminary experiments we find that using rational activation functions can lead to a faster convergence during pre-training as well as a higher performance on several downstream tasks.

In both research and industry, there are multiple factors to consider when comparing models. Our current ML benchmarks measure one aspect of this, e.g. NLI, NER, QA. How do we integrate different aspects of model performance when comparing models?

Understanding clinical outcomes requires to integrate different modalities in a single latent representation. We present such operators that reuse clinical large language models in English language and integrate complementary medical latent representation from low resource languages, from ontologies, from time variant data and from set data. An example application is the differential diagnosis at https://outcome-prediction.demo.datexis.com.

Current NLP models are increasingly larger and data-hungry, which poses important environmental challenges. In this talk, I discuss several ways in which sparsity might lead to more efficient NLP models. The current life cycle of NLP models offers several opportunities to improve memory and runtime efficiency at different stages: during pretraining, during finetuning, and during inference. I first distinguish between model sparsity and activation sparsity. Then, I focus on adaptive sparse attention approaches for the latter, where the softmax transformation is replaced by sparse transformations – entmax – which maintain end-to-end differentiability and have a learnable parameter which controls their sparsity. I finish by asking several open questions and inviting discussion.

Richard Sutton’s essay “The Bitter Lesson” argues that “general methods that leverage computation are ultimately the most effective”. In this talk, I will argue that the bitter lesson implies that, at a given point in time, it is often possible to outperform large-scale methods with methods that are more efficient and clever. Furthermore, actively working to develop more efficient methods has often uncovered new approaches that scale better. I call this perspective “the sweet lesson” and will present many examples of this principle. Finally, I will wrap up with some thoughts on how to internalize bitter and sweet lessons in NLP’s current era of scale.

Some thoughts and new data on community preferences for how papers should be matched to reviewers.

In this talk, I’ll be presenting the BigScience (https://bigscience.huggingface.co) project. A collaborative experiment in building a multilingual large scale dataset as well as a multilingual large language model, inspired by other fields of research like the Large Hadron Collider.

This breakout session concerned questions about what we should measure and report for NLP experiments and how efficiency can be measured. A common problem with current practice in NLP is that efficiency is either not reported at all or that the used metrics are hard to compare. Different hardware environments often further require individual solutions. Shared tasks and benchmarks with fixed hardware were identified as one attempt to mitigate the problem of comparability. The MLPerf benchmarks [1] were mentioned as one positive example. However, participants raised the question whether such benchmarks lead to overfitting and distract from real life concerns. Also, different tasks require different constraints, e.g. the pre-training of a large language model entails different concerns than inferencing on this model. Use cases should therefore be viewed from different perspectives. The group agreed that pushing people to report and review efficiency measures can result in a culture shift and an acceleration of science in general.

The panel, together with contributions from the audience, discussed a number of aspects relating to equity in NLP research. Two of the most prominent discussion items were as follows. (1) An unequal allocation of resources can lead to a misalignment between real-world problems and research work. Sharing of resources (HPC cluster usage, collaborations), and hiring of researchers with experience and passion for real-world problems may provide some mitigation. (2) The lack of diversity in the research community – e.g., geographically and institutionally – can lead to an over-exposure and hype for certain types of work and research agendas. This leaves little attention for progress being made in domains that deviate from the mainstream. Mitigation strategies can include changing the incentive structures for publication, actively endorsing research work on a personal level, promoting the inclusion of researchers in discussions with different backgrounds, and simplifying communication with people affected by real-world problems relating to NLP.