Driving HPC Operations With Holistic Monitoring and Operational Data Analytics

Jim Brandt (Sandia National Laboratories – Livermore and Albuquerque, US,
brandt@sandia.gov)

License: Creative Commons BY 4.0 International license © Jim Brandt
This talk presents work in progress at Sandia National Laboratories. It includes a comprehensive vision on holistic monitoring and feedback as well as the mechanisms currently being used for data collection and feedback. Approaches to Machine Learning that have been explored are presented along with work in progress for turning signals from a ML engine into feedback to drive response for mitigating specific application performance degrading conditions in an HPC Lustre file system.

Utz-Uwe Haus (HPE HPC/AI EMEA Research Lab – Zurich, CH, utz-uwe.haus@hpe.com)

License: Creative Commons BY 4.0 International license © Utz-Uwe Haus
Starting from the well-known 4x4 table of characteristics for component vs action levels for MODA (Bates et al.) we highlight the importance and tight interconnection to digital twins of data centers, in particular HPC data centers including System level twins. Recent advances in data-aware middlewares, scheduling simulation tools and high frequency monitoring are discussed as essential components in a holistic, sustainability focused approach.

Michael Ott (Leibniz Supercomputing Centre – Garching, DE, ott@lrz.de)

License: Creative Commons BY 4.0 International license © Michael Ott
User applications very often are black boxes for HPC operators in terms of requirements and runtime characteristics. Yet, having this information about user workloads would be beneficial for multiple purposes: procurement of new systems that better fit the user requirements, influencing scheduling decisions to optimize system throughput, or identify jobs with low performance. This talk will present ideas on continuous performance counter sampling to obtain better insight into HPC workloads without user interaction.

Kevin Huck (University of Oregon – Eugene, OR, US, khuck@cs.uoregon.edu)

License: Creative Commons BY 4.0 International license © Kevin Huck
From the application performance measurement perspective, limited to the user space domain, there are several motivations for runtime monitoring of the application, operating system and hardware. These motivations run the gamut between simple curiosity to identifying the causes of software failures. In this talk, we will explore these motivations and discuss the barriers to getting access to exposed as well as privileged data to contribute to performance understanding.

Esa Heiskanen (CSC – IT Center for Science – Kajaani, FI, esa.heiskanen@csc.fi)

License: Creative Commons BY 4.0 International license © Esa Heiskanen
Installing and operating large HPC systems is complicated from an operational point of view. There are always unpredictable things that happen and those challenges and risks can be mitigated with good preparation and design of different elements. Having better visibility and predictability to system and applications helps operators to understand the workload of a system and perform preventive anomaly detection and maintenance. Digital twins are defined in 5 levels as descriptive, informative, predictive, comprehensive and autonomous. Building a digital twin of a data center and HPC system is challenging and needs co-operations with multiple parties to achieve goals that give real benefits of digital twin.

Taylor Groves (NERSC – Berkeley, US, tgroves@lbl.gov)

License: Creative Commons BY 4.0 International license © Taylor Groves
The focus of NERSC is moving from simulations limited to a single site to complex workflows that span multiple sites – incorporating both data center and HPC technologies. This patchwork of telemetry and control creates additional challenges for data analysis. In this lightning talk, I present the perspective from NERSC and highlight challenges and opportunities for future research.

Terry Jones (Oak Ridge National Laboratory – Knoxville, US, trjones@ornl.gov)

License: Creative Commons BY 4.0 International license © Terry Jones
For many years, High Performance Computing (HPC) relied on performance improvements based chiefly on frequency scaling; during that era, performance portability was straightforward between different generations of machines. Current computer architecture trends have shifted to other strategies for improvements and exhibit rapidly increasing complexity and extreme heterogeneity to provide performance gains. Codesign activities have emerged as vital processes for mapping applications to the underlying hardware. This talk covers key challenges that arise for holistic monitoring when trying to support codesign.

Tapasya Patki (Lawrence Livermore National Laboratory – Livermore, US, patki1@llnl.gov)

License: Creative Commons BY 4.0 International license © Tapasya Patki
Low-level power and performance dials vary significantly from vendor-to-vendor, reducing the portability of system software and monitoring tools. This talk presents Variorum, a library for vendor-neutral power management, and its integration with the Lightweight Distributed Metric System (LDMS).

Francieli Boito (University of Bordeaux/Inria – Bordeux, FR, francieli.zanon-boito@inria.fr)

License: Creative Commons BY 4.0 International license © Francieli Boito
Existing options for I/O characterization on HPC systems include the creation of complete traces – a log of all I/O operations and their information – or of aggregated statistics, such as the total number and size of accesses. In this talk, I discuss the importance of temporal I/O behavior information and recent advances in using signal processing techniques to obtain it and represent it in a concise manner.

Mike Showerman (NCSA – Urbana, US, mung@illinois.edu)

License: Creative Commons BY 4.0 International license © Mike Showerman
Job anomaly detection allows for identifying irregular job behavior that may affect job execution times, other user’s job performance, or system throughput. Examples include load imbalances, stalled jobs, unused requested resources (like GPUs), or high shared resource usage. This talk presents the container-based job anomaly detection that was deployed on the Blue Waters system at NCSA. It allowed for analyzing thousands of jobs in real time on a 27,000 nodes HPC system.

Ivy Peng (KTH Royal Institute of Technology – Stockholm, SE, ipeng@acm.org)
Kathleen Shoga (Lawrence Livermore National Laboratory, US, shoga1@llnl.gov)

License: Creative Commons BY 4.0 International license © Ivy Peng, Kathleen Shoga
The Sonar monitoring infrastructure is a central system-monitoring infrastructure deployed at Livermore Computing. Its main components include sample monitoring, data ingestion, staging, archive, and analytics. In this work, we showcase the analysis results from Sonar monitoring infrastructure that can drive the system co-design for the memory subsystem on next generation HPC systems. Memory subsystem is one crucial component of a computing system. Co-designing memory subsystems becomes increasingly challenging as workloads continue evolving on HPC facilities and new architectural options emerge. This work provides a large-scale study of memory utilization with system-level, job-level, temporal and spatial patterns on a CPU-only and a GPU-accelerated leadership supercomputer. From system-level monitoring data that spans three years, we identify a continuous increase in memory intensity in workloads over recent years. We showcase how monitoring data on production systems can reveal different hotspots in memory usage in applications on large-scale systems. We introduce two metrics, “spatial imbalance” and “temporal imbalance”, to quantify the imbalanced memory usage across compute nodes and throughout time in jobs. Finally, we identify representative temporal and spatial patterns from real jobs, providing quantitative guidance for research on efficient resource configurations and novel architectural options.

Phil Carns (Argonne National Laboratory – Lemont, US, carns@mcs.anl.gov)

License: Creative Commons BY 4.0 International license © Phil Carns
Methods of measuring and recording performance data have been around as long as computers themselves, and the first reaction to any performance data is to imagine how to improve performance. Why isn’t this more automated by now? This talk will share my experiences dealing with this problem in the arena of HPC I/O, including challenges in identifying trigger behaviors, selecting responses, and mechanisms for enacting responses. Thoughtful solutions to these problems have the potential for enormous positive impact in scientific computing.

Frédéric Suter (Oak Ridge National Laboratory – Knoxville, US, suterf@ornl.gov)

License: Creative Commons BY 4.0 International license © Frédéric Suter
Uncertainty in job duration, caused by different input parameters or resource sharing can cause a loss of work if the resource reservation is not well dimensioned. To address this issue, several approaches can be followed at different times. Simulation can be used as a comprehensive twin of the job and help to improve the estimation of the job duration before its submission. Uncertainty-aware scheduling strategies can be exploited by the job and resource management system which is the only system component that has a global view of all the jobs and the resources. Finally, by running a job co-pilot that can access both the performance data exported by the application and information coming from the job scheduler, we can better react to unexpected events and prevent unwanted loss of work.

Valeria Cardellini (University of Roma Tor Vergata, IT, cardellini@ing.uniroma2.it)

License: Creative Commons BY 4.0 International license © Valeria Cardellini
To tackle the fundamental challenges of performance and workload uncertainty, we can exploit a hierarchical control architecture to adapt at runtime the managed system or application. This control pattern avoids the scalability limitations of fully centralized controllers as well as the lack of coordination of fully decentralized schemes. Moreover, it allows for separation of concerns and time scales. As a case study, I present how to manage the horizontal auto-scaling of distributed applications deployed over heterogeneous computing infrastructures by means of heterogeneity-aware adaptation policies that run on a two-layered hierarchy of controllers. Application-level controllers steer the adaptation process for whole applications, aiming to guarantee user-specified Quality of Service requirements while easing the setting of control knobs that are exposed to the users. Lower-layer controllers take auto-scaling decisions for single application components using reinforcement learning techniques. To adopt the latter in an efficient and beneficial way, we can blend together models and learning to improve the learning velocity as well as function approximation techniques to reduce the memory requirements.

Ann Gentile (Sandia National Laboratories – Albuquerque, US, gentile@sandia.gov)

Main reference: SAND2023-02343PE

License: Creative Commons BY 4.0 International license © Ann Gentile
Operational efficiency can be gained through data-driven operations, intelligent resource management, and resiliency handling, however the current data-gathering, communication, and response hooks do not exist! We propose that a fundamental redesign of system software, middleware, applications, and architectures and their interactions is needed to provide the hooks and all the components must be self-aware and self-organizing in order to make the right local and global decisions on the time-scales necessary for effective response.

Abdullah Mueen (University of New Mexico – Albuquerque, US, mueen@unm.edu)

License: Creative Commons BY 4.0 International license © Abdullah Mueen
HPC performance monitoring tasks must respond back to the system and users in order to improve the overall performance, usability, resilience, robustness among many others. Data mining techniques can serve these needs by catering real-time knowledge extraction tools. In this talk, I show streaming data mining tools for pattern matching, clustering and anomaly detection that are data agnostic and applicable to performance monitoring data. I also discuss the potential response mechanisms that can be supported by offline knowledge extraction and online learning techniques with inputs from applications.

Devesh Tiwari (Northeastern University – Boston, US, d.tiwari@northeastern.edu)

License: Creative Commons BY 4.0 International license © Devesh Tiwari
Traditionally, we have assumed that large-scale computing users are fairly boring and that their workloads often do similar things repetitively. But, now things are changing and changing fast. Our workloads and users are becoming interesting and, often, are surprising us with new trends and behavior. In this talk, I will discuss a few lessons I learned as we applied AI/ML methods to HPC resource management problems. In particular, I will discuss performance variability identification, performance auto-tuning, and open-sourcing a large HPC datasource.

Krishnan Raghavan (Argonne National Laboratory – Lemont, US, kraghavan@anl.gov)

License: Creative Commons BY 4.0 International license © Krishnan Raghavan
In this talk, I present the analysis challenges that we face when applying graph neural networks in detecting workflow anomalies. I begin this talk with an overview on graph neural networks and describe the need for them. I then discuss the challenges that are encountered when modeling the data from a workflow as graphs and illustrate the notion of imbalance and noise in this context. I then define how these challenges amplify when more and more data is collected from the system. Moreover, to correct this issue, I illustrate the need for continual learning in this problem and end the talk with the question, “how can we model and correct the challenges due to the drift in the distribution of the data?”

Thaleia Doudali (IMDEA – Madrid, ES, thaleia.doudali@imdea.org)

License: Creative Commons BY 4.0 International license © Thaleia Doudali
In this talk, I will share some lessons learned on how we can build practical foundations when designing hybrid memory management systems that leverage machine learning methods to improve their effectiveness. Then, I will make a case on how the integration of visualization inside the systems software has potential to accelerate and improve resource management systems. I will end with a crazy idea on how changing data representations into image or text can unlock new opportunities for integrating cross-domain algorithms into systems solutions.

Hilary Egan (NREL – Golden, US, hilary.egan@nrel.gov)

License: Creative Commons BY 4.0 International license © Hilary Egan
In this talk I present a series of case studies in attempting to use AI/Analytics to improve the sustainability in computing at NREL’s ESIF data center, and some future needs and directions for AI algorithms in this space. I begin with an case anomaly detection for advanced cooling systems study done in collaboration with HPE, showing examples on a blower door failure and scale build-up in a heat exchanger. Next I show attempts to optimize system PUE with facility set points and argue we need to integrate digital twin modeling. I then argue a wide variety of grid-integration technologies will require sophisticated load shaping and present preliminary results in this space. I conclude with discussing future directions in AI including digital twins, modular AI, time-series foundation models.

Jeff Hanson (Hewlett Packard Enterprise – Spring, US, jeff.hanson@hpe.com)
Torsten Wilde (Hewlett Packard Enterprise – Böblingen, DE, torsten.wilde@hpe.com)

License: Creative Commons BY 4.0 International license © Jeff Hanson, Torsten Wilde
Available data for HPC data centers are increasing in velocity and scale. Using traditional threshold based alerting mechanisms or visual inspection of operation graph data are limited in scope and resolution of detection. In this talk we present an implemented framework that uses ML/AI models to work on streaming data to detect anomalous behavior and reduce alarm fatigue for the system administrators. Enabling the research community to develop and implement control loops based on system information is not easy since system integrators provide their own management frameworks to keep the system in a defined and safe operating state. In the second part of this talk we will highlight the challenge using system power and energy management as an example. We will propose different approaches that would enable community developed management and control processes to work in synergy with system integrator provided mechanisms.

Oriol Vidal (Barcelona Supercomputing Center, ES, oriol.vidal@bsc.es)

License: Creative Commons BY 4.0 International license © Oriol Vidal
Full power management of large infrastructures requires an autonomous way to monitor system running jobs, as well as identifying those power saving opportunities. This talk will present how the EAR software fills ODA framework’s gaps concerning HPC power and energy management, from basic monitoring and accounting to power saving strategies during application runtime and cluster-level power-cap.

Isaías A. Comprés Ureña (TU München – Garching, DE, compresu@in.tum.de)

License: Creative Commons BY 4.0 International license © Isaías A. Comprés Ureña
The priority-based First-Come First-Serve (FCFS) with backfilling heuristic has been very successful in maximizing node utilization. However, since node utilization depends on system state and user provided jobs that are queued, it is not always possible to reach close to full node utilization. We propose malleability as a means to add additional opportunities for schedulers to reach near full node utilization. With malleability, nodes can be added or removed from jobs. Monitoring and modeling techniques are needed, to ensure quality decisions are made.

Sven Karlsson (Technical University of Denmark – Kongens Lyngby, DK, svea@dtu.dk)

License: Creative Commons BY 4.0 International license © Sven Karlsson
Since the improvement of computer performance can no longer be satisfied by simply raising clock frequencies, architectures are evolving towards both the multiplication of processing elements and heterogeneity of their functions. The aforementioned execution platforms combined with steadily exacting non-functional performance requirements such as execution speed, timeliness, and power consumption challenges traditional design methods. The PaPP project included work on many aspects including improved resource management, adaptivity in all parts of the technology stack including the applications, and a control loop for managing non-functional performance requirements. The PaPP software infrastructure and the control loops were briefly introduced with demonstrations presented.

Thomas Jakobsche (University of Basel, CH, thomas.jakobsche@unibas.ch)

License: Creative Commons BY 4.0 International license © Thomas Jakobsche
HPC systems provide high computing power to different domain scientists. Not all users possess the necessary knowledge and experience to fully utilize the capabilities of HPC systems. Certain users tend to submit inaccurate resource requests, leading to inefficient job scheduling, longer job wait times, and ultimately to wasted computing resources. HPC systems offer several tools to support users in order to improve their resource requests (e.g. SLURM’s seff and sacct commands). However, these commands are missing easily digestible job statistics that can provide, for example, the longest execution time for a group of jobs. This lightning talk presented a command-line functionality based on the SLURM sacct command that introduces grouping and tabular-like functionality to job accounting data, that is easily available to administrators and users. This tool can support administrators and users to quickly identify groups of jobs with inaccurate resource requests, create job efficiency reports to raise awareness, and support users to improve future resource requests based on previous job submissions.

Thomas Gruber (Universität Erlangen-Nürnberg, DE, thomas.gruber@fau.de)

License: Creative Commons BY 4.0 International license © Thomas Gruber
Applications for ODEs and PDEs often benefit from different coder variants but performance is highly dependent on the input. For testing these variants as well as hardware and runtime parameters different combinations need to be tested. By letting a genetic algorithm select testable parameter combinations, the amount of combinations can be reduced to a reasonable set. With measurements, the combinations can be graded and the best one selected for the given input. With refinements from run to run, the application can be executed with little disturbance and still find optimal execution settings. The user interacts with the library by supplying code variants through code instrumentation.

Jim Brandt (Sandia National Laboratories – Livermore, and Albuquerque, US,
brandt@sandia.gov)

License: Creative Commons BY 4.0 International license © Jim Brandt
This presentation was about an explicit use case at Sandia National Laboratories where processes of an application can be handed Lustre Object Store Targets (OSTs) with significantly poorer performance than other processes. The worst performance gates the application performance and in the presented case significantly increases the application run time (e.g., >50%). The mitigating approach being explored is creation of a feedback loop from the monitoring and analysis system to the application which could then take the action of deleting and re-creating the file it is writing to and ideally be provided with more performant OSTs.