Resource-Efficient Machine Learning

Oana Balmau (McGill University, CA oana.balmau@cs.mcgill.ca)
Matthias Boehm (TU Berlin BIFOLD, DE, matthias.boehm@tu-berlin.de)
Ana Klimovic (ETH Zürich, CH, aklimovic@ethz.ch)
Peter Pietzuch (Imperial College London, GB, prp@imperial.ac.uk)
Pınar Tözün (IT University of Copenhagen, DK, pito@itu.dk)

While the capabilities of machine learning models have become more and more impressive in the last decade, one cannot overlook the computational footprint of their end-to-end lifecycle. According to the Stanford AI Index Report [52], the computational complexity of the state-of-the-art language models has increased 7 orders of magnitude since 2017. In turn, this increases the estimated costs to train these models in the cloud by 5 orders of magnitude, and the carbon footprint of training such models are equivalent to 10s of human years. Furthermore, the cost of training is only a fraction of the whole costs. After training, then comes the cost of continuously deploying these models, which depend on the way these models are used for inference and the frequency of retraining to update the models.

The participants of the Dagstuhl Seminar on Resource-Efficient Machine Learning (ML) targeted the computational efficiency challenges for machine learning, especially deep learning, from different angles and by focusing on the different stages. On the first day of the seminar, we split into four groups, each with a specific focus. The groups identified the research questions they want to focus on, delved deeper into the existing work, and identified future steps to continue collaborations beyond the seminar.

The first group, Resource-Efficient Data Selection (Section 3), targets the efficiency of data selection methods for training deep learning models. Data selection is a preliminary step before any model training, but specifically for fine-tuning tasks, where a pre-trained model must be specialized for a specific task. The effectiveness of a data selection method is typically evaluated by the accuracy it achieves for the given task. The assumption is, as a side effect, if one can achieve a certain accuracy while using less data, this would improve the efficiency of training. This group questions this assumption and asks the following research question: what are the trade-offs between the computational complexity of a data selection method, its effectiveness in terms of model accuracy, and the end-to-end training efficiency?

The second group, The Future of Portable, Extensible, and Composable Machine Learning Systems (Section 4), aims at making the emerging ML systems support a larger diversity of applications more efficiently. At the core of this support lies a departure from the dominant reliance on dense tensor computations. Targeting a larger diversity in applications also requires looking at a larger variety of hardware devices, beyond large accelerators that are highly optimized for dense matrix computations. Targeting such diversity requires co-design and finding the right abstractions across software, compilers, and hardware. The group’s vision results in several research challenges with the following overarching research question: how to design holistic and composable software and hardware frameworks for ML?

The third group, Hardware-Software Co-Design for Machine Learning (Section 5), target similar research challenges to the second group, but with a deeper focus on hardware diversity. The group identifies that the conventional way of optimizing machine learning tasks for a certain hardware device is through tight coupling between high-level ML engineering and low-level performance optimizations. Certain high-level optimizations, with a specific hardware in mind, in turn, hinders portability to different hardware devices, resulting in sub-optimal efficiency and missed opportunities for functionality. Therefore, the key research question here is how does one create a hardware stack for ML that enables better portability across different devices?

The fourth group, Workload-Aware Machine Learning Serving (Section 6), focus on ML serving. Serving ML models, especially large language models (LLMs), at scale is highly costly and requires substantial hardware resources. To achieve more resource- and performance-efficient model ways of serving models, one needs to adaptively determine which specialized model to serve or cache, or how to optimize a model. This adaptivity is highly dependent on the workload needs that may be dynamic. This group, therefore, aims to answer the following questions: (1) what is the behavior and needs of the real-world serving workloads and (2) how does one build a framework that enables adaptive model serving based on dynamic user and workload needs?

In the following paragraphs, we briefly introduce the state-of-the-art data selection works, and based on our discussions at the Dagstuhl Seminar, outline our vision on how to make data selection more resource-efficient.

In that context, we propose and discuss a vision of future ML systems, perhaps in 5–10 years from today. We expect such systems will go beyond the current focus on dense, tensor arithmetic and incorporate aspects of wider computer science and high-performance computing calculations. As examples of the kinds of richness the future might hold, we draw inspiration from what we call the Four Languages taxonomy that spans four different algebras:

• $\mathrm{■}$

  Tensor algebras work on tensors (linear relationships between sets of objects) expressed as an array of values as supported by modern ML frameworks [8, 43] and numerical systems [11, 53].

• $\mathrm{■}$

  Relational algebras work on relations between objects (a subset of the Cartesian combination of one or more sets) as supported by relational database management systems [18, 28, 40, 45, 50].

• $\mathrm{■}$

  Graph algebras work on edge relationships between sets of objects (such as knowledge graphs, social network graphs, and meshes) as supported by graph engines [3, 36].

• $\mathrm{■}$

  Spatial algebras work on metric spaces that express distance relationships between sets of objects as supported by graphics rendering engines [4], molecular dynamics packages [1, 2] and ML-based recommender models.

We base our vision on three suppositions regarding the future of ML systems. We believe these suppositions to be likely, given past trends in the ML computing space:

1. 1.

   ML applications will be exciting and diverse. ML models will not be limited to dense tensor computations composed in different ways. Instead, we expect them to evolve in unexpected ways, to incorporate components from the four algebras above. Perhaps, this involves increased reliance on hierarchies, irregular and sparse models, spatial embeddings, and computation with irregular data models such as relational models and graphs.

2. 2.

   Hardware will be exciting and diverse. Hardware will be unexpected in various ways, and not limited to CPUs, programmable GPUs, and dense vector or matrix units. As ML models increase in complexity and embrace components from different algebras, we will see hardware to cater to those algebras. In addition, we will increasingly find ways to build adaptation and reconfigurability into hardware substrates, perhaps through programmable networks, configurable collection-oriented operations, and through support for different data-types and precision.

3. 3.

   Computation will be distributed across centers and edges. We also expect that this heterogeneity goes far beyond the data center focus of current large language model (LLM) training and inference systems to incorporate a wide range of edge, mobile, internet of things (IoT), and other deployment scenarios. Each of these areas will have different economies and trade-offs among cost, compute and memory performance, power (both dissipated and battery capacity), and latency/bandwidth/reliability of communication links.

With these many dimensions of heterogeneity, we imagine a complex, potentially fully connected, graph of interfaces between components with various diverse strengths and capabilities, even potentially changing at runtime by adaptation and reconfigurability to deal with changing requirements and environments. Components (which combine hardware accelerators, software stacks, runtimes, and application software) may have to talk to other components that speak a different language, which makes each interface an opportunity for co-design optimization across the languages.

Each of the languages can express many of the computations of the others. However, each language is conceptually very different, lends itself to a different style of thoughts, and comes with some degree of induced awkwardness or structure in expressing particular computations. And similarly for the reductions or translations from one language to another (as is typical of theoretical reductions). But the structure of these equivalences and reductions induces interesting trade-offs and opportunities for cross-domain optimization and expressiveness. Such fused systems might have huge opportunities for whole-system efficiency. But to realize those opportunities, we need to build first the individual components, then the interfaces between them, and lastly the entire system in a way that allows us to move work fluidly across and into these future ML computing systems.

We do not yet have a strong enough signal to recommend embarking on a four languages commercial systems project. Due to the involved complexity this could be a too aggressive jump towards a single grand unified theory of four very different styles of computation. However, we see a range of research challenges to approach this vision:

1. 1.

   How to design a holistic and composable framework at the software and hardware level?

2. 2.

   How to design user-facing language abstractions and a system-internal unified intermediate representations for combining the different data models and algebras?

3. 3.

   How to do transformations across the different data representations (i.e., tensor, relational, graph, spatial) enabling portability across the different hardware substrates?

4. 4.

   How to design the interfaces between these components and how to ensure efficient data exchange?

5. 5.

   How to assign components to different heterogeneous compute architectures such as CPUs, GPU, TPUs, or even more specialized substrates, and use heterogeneous communication links between them?

6. 6.

   How to compose domain-specific hardware blocks and at what degree of programmability?

7. 7.

   How to make hardware reconfigurable and adaptable at run-time, and exploit such adaptability throughout the framework (i.e., language, compiler, runtime)?

8. 8.

   How to allow experts to fine-tune the system by extending it with hand-written code where automatic approaches are still missing or insufficient?

To conclude, we think there is value in the vision of uniting the four languages and the rich, effective, and expressive future for ML that such a unification might support. Even if we only partially unify some of the areas (as physics did with the electromagnetic and weak forces), it could still be valuable to ML in both theory and practice.

The current ML stack is structured across multiple levels. At the top, an ML engineer writes code in a high-level framework such as PyTorch [5] or Tensorflow [6], defining an abstract dataflow graph. To achieve high performance, the high-level graph undergoes a compilation process, involving transformations at multiple levels (in the form of Intermediate Languages [7]), that target a specific hardware setup (i.e. $N$ machines of type $X$). High-level optimizations such as operator fusion are applied first, followed by low-level, hardware-dependent operations such as buffer materialization and padding [60, 13, 46]. As a result, the compiler outputs operators and libraries that target only the specific hardware setup.

Although the current compilation process leads to high performance, we identify some key issues that make it incompatible with the current and future landscape of heterogeneous ML models and hardware. First, we observe an excessive coupling between high-level ML engineering and low-level performance optimization, hindering portability, both for functionality and performance. ML engineers often perform premature optimization at the model definition stage to maximize efficiency, without having full knowledge of the hardware and its attributes (for example deciding on a specific memory layout or operator fusing). However, fully decoupling high-level model definitions from low-level implementation and optimizations is impractical since the compiler needs some guidance to navigate the large search space [13] and discover efficient plans in a reasonable time. Additionally, as ML is moving at such a fast pace, the current ML stack fails to keep up with the recent advances in a principled way, compromising correctness and soundness. Finally, the tight coupling between ML model definition and hardware setup, and the focus on real-world performance, makes us unable to reason about performance at different scales and limits access for academic researchers.

We propose a principled redesign of the ML software stack and identify key research questions that need to be addressed at each level. The proposed ML stack is illustrated in Figure 1. At the top level, ML engineers focus on abstract model design (i.e. mathematical formulation) without considering hardware specifications. A critical first step here is defining the input at this high level (i.e. is it the whole ML model graph, or several such that interact via some programming constructs). We also argue that there should be restrictions on which high-level operations are supported. Next, as in the current ML stack, multiple transformations are performed. However, instead of targeting a specific hardware setup, we propose targeting an abstract multi-level SPMD execution model, representing a virtual, multi-level Hardware stack. This would allow the compiler to generate “executables” at different levels of abstraction and understand tradeoffs and optimizations at different levels. It would also enable modeling of existing and future hardware in an abstract way, enabling portability in terms of functionality and performance, accelerating the design of new hardware and software in a principled manner, and allowing correctness guarantees at different levels.

For this Hardware stack to be effective, we need to identify what types of information we need to pass to the compiler at different levels. We propose three key levels. At the first level, we define distinct classes of abstract heterogeneous devices, each with a specific memory hierarchy and access granularity, number of compute units, supported functions, and compute-to-memory ratio. At a second level, we view each possible system setup as a collection of these heterogeneous abstract devices, that can communicate with a specific set of primitives. An example of a step in this direction are the abstractions for operations and communication primitives in the CINM and C4CAM [31, 20] multi-level compiler frameworks for emerging near and in-memory accelerators. Finally, the last level handles runtime optimizations, mapping abstract (virtual) to physical devices. At this stage, the compiler decides on specific memory layouts and operator mapping.

In conclusion, we propose reimagining the ML stack in a principled way with explicit consideration of semantics and correctness guarantees across multiple abstract hardware layers. To implement this, it is essential to clearly define interfaces and intermediate representations required at each level of the ML stack, and evaluate the tradeoffs between performance and adaptability in the fast-moving ML space.

Serving machine learning (ML) models at scale, particularly modern large language models (LLMs), is extremely resource-intensive and costly. We observe two major trends that increase resource requirements of ML serving workloads. First, there is increasing scale in terms of number of client requests and model sizes. Increasing context-lengths in conversational LLM interactions also generates large state per client. Second, we see increasing model personalization for different domains and user groups (through fine tuning and prompt engineering), model specialization for different tasks (through custom models), as well as multi-modal and multi-component models. There are, however, new opportunities to mitigate these challenges. Weight pruning and other forms of sparsity exploitation are becoming more practical. Combined with new hardware features for reconfiguration and tuning, these increase the optimization potential for ML serving.

• $\mathrm{■}$

  Matthias Böhm – TU Berlin, DE

• $\mathrm{■}$

  Maximilian Böther – ETH Zürich, CH

• $\mathrm{■}$

  Marco Canini – KAUST – Thuwal, SA

• $\mathrm{■}$

  Jerónimo Castrillón-Mazo – TU Dresden, DE

• $\mathrm{■}$

  Patrick Damme – TU Berlin, DE

• $\mathrm{■}$

  Khuzaima Daudjee – University of Waterloo, CA

• $\mathrm{■}$

  Pamela Delgado – HES-SO Valais-Wallis – Siders, CH

• $\mathrm{■}$

  Thaleia Dimitra Doudali – IMDEA Software Institute – Madrid, ES

• $\mathrm{■}$

  Jens Hagemeyer – Universität Bielefeld, DE

• $\mathrm{■}$

  Steven Hand – Google – Mountain View, US

• $\mathrm{■}$

  Dagmar Kainmüller – Max-Delbrück-Centrum – Berlin, DE

• $\mathrm{■}$

  Fredrik Kjolstad – Stanford University, US

• $\mathrm{■}$

  Ana Klimovic – ETH Zürich, CH

• $\mathrm{■}$

  James Kwok – HKUST – Hong Kong, HK

• $\mathrm{■}$

  Manisha Luthra Agnihotri – TU Darmstadt, DE & DFKI – Darmstadt, DE

• $\mathrm{■}$

  Peter R. Pietzuch – Imperial College London, GB

• $\mathrm{■}$

  Tilmann Rabl – Hasso-Plattner-Institut, Universität Potsdam, DE

• $\mathrm{■}$

  Theo Rekatsinas – Apple – Zürich, CH

• $\mathrm{■}$

  Ties Robroek – IT University of Copenhagen, DK

• $\mathrm{■}$

  Stefanie Scherzinger – Universität Passau, DE

• $\mathrm{■}$

  Tom St. John – Meta – Menlo Park, US

• $\mathrm{■}$

  Foteini Strati – ETH Zürich, CH

• $\mathrm{■}$

  Shinya Takamaeda-Yamazaki – The University of Tokyo, JP

• $\mathrm{■}$

  Pinar Tözün – IT University of Copenhagen, DK

• $\mathrm{■}$

  Lluís Vilanova – Imperial College London, GB

• $\mathrm{■}$

  Eiko Yoneki – University of Cambridge, GB

• $\mathrm{■}$

  Cliff Young – Google – Mountain View, US

• $\mathrm{■}$

  Ehsan Yousefzadeh-Asl-Miandoab – IT University of Copenhagen, DK