2026-07-20T04:47:58Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:26342 2026-07-16T10:50:12Z ddc:004 open_access

Sustainable Benchmarking Tool (Tool Paper) Iser, Ashlin Anastacio, Marie Matricon, Théo Simon, Laurent Hoos, Holger H. Sustainability Empirical performance comparison Benchmarking Problem instance selection Solvers for NP-hard problems from areas such as automated reasoning or optimisation are complex systems in which many different components interact. The performance of these solvers is the result of an intricate interplay between implementation details, algorithmic concepts and heuristics. This, alongside the complexity of the problem instances to be solved, makes it challenging to assess the effect of a single idea on the overall performance of a given solver. It is therefore not only crucial, but also challenging to evaluate the performance impact of new ideas. Existing reliable evaluation methods require large sets of diverse benchmark instances and considerable amounts of computing resources. This makes empirical evaluation a bottleneck for solver development, as it is time-consuming and energy-intensive, often requiring several CPU years of computation to evaluate the impact of a single idea. In recent years, this bottleneck has led to the development of data-driven approaches that can dynamically select a smaller number of instances that provide sufficient statistical evidence to evaluate the relative performance of a given set of solvers. However, these methods are typically not easily accessible. In this work, we present a tool that implements these methods and makes them readily accessible to solver developers, thus enabling them to obtain swifter feedback on their ideas. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Ashlin Iser and Marie Anastacio and Théo Matricon and Laurent Simon and Holger H. Hoos 2026 Is Part Of LIPIcs, Volume 377, 29th International Conference on Theory and Applications of Satisfiability Testing (SAT 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.SAT.2026.36 urn:nbn:de:0030-drops-263427 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.SAT.2026.36 eng https://creativecommons.org/licenses/by/4.0/legalcode