Competitions and Empirical Evaluations in Automated Reasoning (Dagstuhl Seminar 25441)

Abstract

Solver competitions and practical problem solving challenges are a cornerstone in the field of Automated Reasoning (AR). They drive innovation by providing a platform for benchmarking, empirical evaluation, standardization of robust tools and methodologies, and identify challenges from research and industry. These events not only showcase the latest advancements in solver technology but also help to establish best practices for reliability and performance assessment. Organizing such competitions presents significant challenges, including the selection of representative benchmarks, the development of fair evaluation metrics, and ensuring result reproducibility. It is widely acknowledged that continued community engagement is essential for tackling these challenges and strengthening collaboration among organizers, developers, users, and reviewers. This report documents the program and the outcomes of Dagstuhl Seminar "Competitions and Empirical Evaluations in Automated Reasoning" (25441), which centered around competition challenges, discussed questions and solutions with the aim to build a community of practice of competition organization and empirical evaluation in AR.