Automated Machine Learning For Computational Mechanics

This small Dagstuhl Seminar brought together 23 researchers from both engineering and machine learning/optimization, representing both academia and industry. The group included a mix of senior and junior researchers, creating a diverse and collaborative environment.

Over the course of five days, the mornings featured 14 short presentations, each lasting 15 to 20 minutes. The rest of the seminar was organized in a dynamic and flexible manner, with activities ranging from scientific speed-dating and two-way surveys to the traditional trekking, as well as plenary and parallel discussions on topics chosen by the participants (see the complete seminar schedule in Fig. 2). This flexible structure allowed for a more engaging and relaxed schedule, which was appreciated by all attendees. Discussions that began during the day often extended into the evening, where the cozy atmosphere of the Dagstuhl castle played a key role in making everyone feel comfortable during both work-related exchanges and more informal moments.

Based on the survey results, the seminar was widely considered a great success by both the organizers and participants. Despite the participants needing some initial time to get used to the languages of the two different research communities, the seminar successfully bridged the gap between the machine learning and computational mechanics communities, fostering valuable cross-disciplinary dialogue. The event provided a platform for sharing cutting-edge research and insights, while also addressing the practical challenges faced when integrating automated machine learning (AutoML) into computational mechanics workflows.

The presentations were thought-provoking and initiated lively discussions throughout the seminar. The collaborative spirit extended beyond the formal sessions, with participants engaging in productive breakout sessions and working groups. These dynamic exchanges not only explored the current state of research but also laid the groundwork for potential future collaborations.

The event demonstrated the clear potential for synergy between these fields, and it is expected that the connections made during the seminar will continue to grow and lead to impactful advances. The organizers extend their sincere thanks to the Scientific Directorate and the Dagstuhl Center administration and staff for their precious support, which was instrumental in the seminar’s smooth execution and success.

Physics-based machine learning leverages the strengths of both physics-based numerical simulation and data-driven approaches. By combining the flexibility and efficiency of state-of-the-art machine learning (ML) such as deep learning with the rigor of classical continuum mechanical and thermodynamically models and numerical methods, accurate and fast predictions can be obtained in a reliable and robust manner. This hybrid approach opens up great potential for solving the current challenges in computational solid mechanics. The current work introduces feed-forward neural networks that enforce physics in a strong form to tackle computational fracture mechanics problems. Our proposed model undergoes training with various load sequences and is then evaluated for its capacity in both interpolation and extrapolation. This study is the first to explore combining physics-based models and machine learning to address brittle and ductile fracture.

In this tutorial-like presentation, I will motivate AutoML and provide an overview over the major areas inside of AutoML: hyperparameter optimization, neural architecture search, multi-objective optimization, AutoML systems and meta-learning. I will put some focus on Bayesian optimization and the meta-learning of new algortihms in the prior-fitter networks (PFN) framework.

The talk focuses on engineering design optimization problems in industry and the application of AutoML, Hyperparameter Optimization, and Algorithm Selection for solving and handling such tasks. Structural mechanics problems in the automotive industry, such as in car body crash optimization, are discussed as one of the key examples in this application domain. Direct optimization using variants of Evolutionary Strategies and AutoML for response surface modeling, one-shot optimization, and fast, interactive proxies for simulation models are discussed. As computational effort of simulations is a key bottleneck for optimizer selection and configuration, we also present an idea towards using evolutionary landscape analysis features for finding fast proxy functions that can be used for both tasks instead of the original problem. Showing that AutoML can be used for generating response surface models automatically, and proxy functions can be used for algorithm selection and configuration, the key questions really are: How can AutoML be used in practice to support structural mechanics and other engineering design tasks and how can we tackle computationally expensive real-world problems, extract knowledge, and learn from the data generated?

Simulating the deformation of solids is essential to understand the behavior of complex structures and attain competent designs. Accordingly, computational techniques such as the finite element method have shown significant progress over the last decades. However, there are remaining associated challenges such as high computational cost and poor stability. In this talk, I would like to touch on the applicability and usefulness of machine learning (ML) approaches for improving the efficiency and robustness of simulation methods. I will demonstrate different problems where we use ML models trained on either pure computational or hybrid computational-experimental data. I will conclude by discussing the potential future directions for the use of ML in computational mechanics and the actions that can foster the connection between these two fields.

I also discussed the grand challenge of generating software platforms that are useful for non-experts and provided some thoughts for how tools from AutoML might play a role to this end. Finally, I led a – quite lively – open-ended conversation with the group on these topics.

Participants: Niels Aage, Paolo Ascia, Carola Doerr, Olaf Mersmann, Marc Zöller, Elena Raponi

The discussion focused primarily on Exploratory Landscape Analysis (ELA). While the extraction of ELA features provides valuable insights into function properties, there are inherent limitations when evaluating these features in real-world problems compared to synthetic benchmarks or analytical functions. The importance of ELA was highlighted, particularly in showing that a small set of relevant features can correlate with performance, which may also be applicable to computational mechanics. In computational mechanics, the approach often involves using approximations or simplifications to achieve characteristics that can be optimized.

Another key topic was preconditioning for specific problems, where certain classes have shown optimality for specific preconditioning methods. Additional considerations included the time dependency of characteristics, hierarchical algorithm selection, and the limited number of solvers available in computational mechanics compared to those used in ELA. Other significant points of discussion included the potential of transfer learning within a medium-dimensional characteristic space (e.g., 20 dimensions) and the comparison between deep ELA features and manually designed ones. For design problems, standardized benchmarks – such as those used in automotive crash tests – were identified as essential. The role of human-machine interaction was also emphasized, particularly in delivering optimal solutions, ensuring diversity, and maintaining interactivity.

The discussion also addressed the optimization of benchmarks and the scarcity of real-world data and functions, raising important questions about the design and future implications of benchmarks. In terms of geometric features, the importance of voxel-based representation was underscored, along with the transformation of these into vector representations with minimal parameters.

Participants: Thomas Bäck, Monica Capretti, Frank Hutter, Melvin Leok, Niki van Stein

The session emphasized the practical aspects of deploying machine learning models in engineering applications, focusing on efficiency, adaptability, and the integration of advanced optimization techniques. More specifically, the following points have been addressed:

1. 1.

   Integration into Production Pipelines:

   • $\mathrm{■}$

     Emphasis was placed on integrating algorithms into production pipelines to provide quick and efficient solutions, contrasting with the academic approach that focuses on achieving the best possible result regardless of time constraints.

2. 2.

   Neural Architecture Search (NAS):

   • $\mathrm{■}$

     Discussion on the potential and challenges of NAS, including the importance of defining proxies to optimize the search process.

   • $\mathrm{■}$

     Mention of a benchmark conducted with Google on NAS and other NAS benchmarks.

3. 3.

   Meta-models for Optimization:

   • $\mathrm{■}$

     The utility of meta-models for algorithm development and selection was highlighted, with potential applications across different domains like crash and composite materials.

4. 4.

   Efficient Model Configuration:

   • $\mathrm{■}$

     The session covered methods for building and configuring meta-models, leveraging machine learning to handle various problem characteristics.

5. 5.

   Hardware Considerations:

   • $\mathrm{■}$

     The role of hardware in optimizing neural networks, including techniques like pruning and quantization to reduce computational costs and improve efficiency.

6. 6.

   Bayesian Optimization:

   • $\mathrm{■}$

     Exploration of Bayesian optimization approaches for parallel evaluations, emphasizing its efficiency in handling expensive data evaluations.

7. 7.

   Dimensionality Reduction:

   • $\mathrm{■}$

     Techniques to reduce dimensionality, such as using auto-encoders and PCA, were discussed to speed up the optimization process (BO approach).

8. 8.

   Generative Models for Data Simulation:

   • $\mathrm{■}$

     The potential of using generative models to simulate datasets that reflect real-world data, including the importance of considering physical constraints and causal relationships in data generation.

9. 9.

   Learning and Updating Models:

   • $\mathrm{■}$

     The session delved into the concepts of learning from prior data, updating models with new data, and the use of pre-trained models for inference.

10. 10.

    Future Directions:

    • $\mathrm{■}$

      Participants discussed the future of optimization methods, including leveraging transformer models for basic inference and exploring general large language models for optimization tasks.

Participants: Fadi Aldakheel, Elsayed Saber Elsayed Ibrahiem Elsayed, Helen Fairclough, Roman Garnett, Alicia Kim, Lars Kotthoff, Peter Krause, Charles Mish, Markus Olhofer, Lisa Pretsch, Thiago Rios, Gokhan Serhat

When AutoML tools are applied to solve a CoMe problem, known equations and experimental results must be formatted appropriately for AutoML. While the CoMe community can easily recognize similarities between tasks, AutoML is particularly valuable in unfamiliar situations. However, AutoML should not be used when an efficient solution is already known. In unfamiliar cases, AutoML tools could benefit from being more interactive, leveraging the knowledge from CoMe to reduce computational time, narrow the search space, and enhance overall performance.

While the CoMe community has a strong understanding of the physical properties of a system, it lacks familiarity with the properties of AutoML tools. As a result, practictioners and users of AutoML tools often struggle to grasp the issues related to the quality of the data provided, how to account for imprecisions in collected data, and how different features are utilized by AutoML tools. Additionally, the CoMe community remains uncertain about which AutoML tools to use under specific conditions and how to properly format the data. In conclusion, both sides acknowledged a clear lack of communication and highlighted that bridging the gap between domain-specific knowledge and automated tools requires close collaboration between researchers from the two fields.

Participants: Niels Aage, Fadi Aldakheel, Carola Doerr, Elsayed Saber Elsayed Ibrahiem Elsayed, Helen Fairclough, Roman Garnett, Frank Hutter, Alicia Kim, Lars Kotthoff, Peter Krause, Melvin Leok, Olaf Mersmann, Charles Mish, Markus Olhofer, Gokhan Serhat, Niki van Stein, Marc Zöller

The session on explainability focused on how AutoML can be leveraged to help the CoMe community better understand their datasets and features. From this perspective, the AutoML community inquired about what the CoMe community is most interested in discovering within their datasets. It was suggested that while AutoML tools cannot independently evaluate the quality of a solution or the problem formulation, they can provide valuable insights into:

• $\mathrm{■}$

  Sensitivity;

• $\mathrm{■}$

  Active constraints;

• $\mathrm{■}$

  Uncertainty;

• $\mathrm{■}$

  Model quality;

• $\mathrm{■}$

  Solution reliability;

• $\mathrm{■}$

  Internal relationship between constraints and variables;

• $\mathrm{■}$

  Contrastive explanation based on what ML considered;

• $\mathrm{■}$

  Visualized data;

• $\mathrm{■}$

  Landscape of the optimization problem;

• $\mathrm{■}$

  Similarity between different solutions;

• $\mathrm{■}$

  Feasible spaces when the ML model is trained on both feasible and nonfeasible points;

• $\mathrm{■}$

  Aggregation of multiple explanations belonging to different model parts.

Participants: Paolo Ascia, Thomas Bäck, Monica Capretti, Lisa Pretsch, Elena Raponi, Thiago Rios

In this discussion, the participants agreed on the properties a CoMe benchmark for the AutoML community should look like. These are:

• $\mathrm{■}$

  A benchmark should be fast to evaluate without the need for any commercial solver;

• $\mathrm{■}$

  The optimal solution for the benchmark should be known and clearly documented for comparison purposes;

• $\mathrm{■}$

  To facilitate rapid testing, the benchmark should include both the computational model and a response-surface-based approximation for quicker assessments;

• $\mathrm{■}$

  The problem posed by the benchmark should meet the following criteria:

  • –

    High dimensionality to reflect real-world complexity;

  • –

    The ability to include constraints, allowing for more realistic and practical scenarios;

  • –

    A non-linear/multimodal/discontinuous landscape to represent typical challenges encountered in computational mechanics;

  • –

    Flexibility to handle either static or dynamic load cases, as required by the context;

  • –

    It should belong to at least the domain of solid mechanics and/or fluid dynamics;

• $\mathrm{■}$

  The benchmark problem should be fully characterized, providing all necessary details for reproducibility and analysis;

• $\mathrm{■}$

  Each benchmark should clearly specify its intended target audience, ensuring it is relevant to the appropriate users;

• $\mathrm{■}$

  Ideally, the benchmark should be derived from well-known simulation software tutorials, as these are familiar and widely recognized by the CoMe community.