Document Open Access Logo

A Framework for Generating Informative Benchmark Instances

Authors Nguyen Dang , Özgür Akgün , Joan Espasa , Ian Miguel , Peter Nightingale

PDF
Thumbnail PDF

File

LIPIcs.CP.2022.18.pdf
  • Filesize: 1.06 MB
  • 18 pages

Document Identifiers

Author Details

Nguyen Dang
  • School of Computer Science, University of St Andrews, UK
Özgür Akgün
  • School of Computer Science, University of St Andrews, UK
Joan Espasa
  • School of Computer Science, University of St Andrews, UK
Ian Miguel
  • School of Computer Science, University of St Andrews, UK
Peter Nightingale
  • Department of Computer Science, University of York, UK

Funding

This work uses the Cirrus UK National Tier-2 HPC Service at EPCC (http://www.cirrus.ac.uk) funded by the University of Edinburgh and EPSRC (EP/P020267/1).
  • Dang, Nguyen: is a Leverhulme Early Career Fellow.
  • Miguel, Ian: supported by EPSRC EP/V027182/1.

Cite AsGet BibTex

Nguyen Dang, Özgür Akgün, Joan Espasa, Ian Miguel, and Peter Nightingale. A Framework for Generating Informative Benchmark Instances. In 28th International Conference on Principles and Practice of Constraint Programming (CP 2022). Leibniz International Proceedings in Informatics (LIPIcs), Volume 235, pp. 18:1-18:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2022)
https://doi.org/10.4230/LIPIcs.CP.2022.18

Abstract

Benchmarking is an important tool for assessing the relative performance of alternative solving approaches. However, the utility of benchmarking is limited by the quantity and quality of the available problem instances. Modern constraint programming languages typically allow the specification of a class-level model that is parameterised over instance data. This separation presents an opportunity for automated approaches to generate instance data that define instances that are graded (solvable at a certain difficulty level for a solver) or can discriminate between two solving approaches. In this paper, we introduce a framework that combines these two properties to generate a large number of benchmark instances, purposely generated for effective and informative benchmarking. We use five problems that were used in the MiniZinc competition to demonstrate the usage of our framework. In addition to producing a ranking among solvers, our framework gives a broader understanding of the behaviour of each solver for the whole instance space; for example by finding subsets of instances where the solver performance significantly varies from its average performance.

Subject Classification

ACM Subject Classification
  • Theory of computation → Constraint and logic programming
Keywords
  • Instance generation
  • Benchmarking
  • Constraint Programming

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Supplementary Materials

References

  1. essence modelling pipeline:. https://constraintmodelling.org/. Google Scholar
  2. Google OR-Tools, 2021. Available from https://github.com/google/or-tools. Google Scholar
  3. Özgür Akgün, Nguyen Dang, Ian Miguel, András Z Salamon, Patrick Spracklen, and Christopher Stone. Discriminating instance generation from abstract specifications: A case study with CP and MIP. In International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research, pages 41-51. Springer, 2020. Google Scholar
  4. Özgür Akgün, Nguyen Dang, Ian Miguel, András Z Salamon, and Christopher Stone. Instance generation via generator instances. In International Conference on Principles and Practice of Constraint Programming, pages 3-19. Springer, 2019. Google Scholar
  5. Ozgur Akgun, Ian P. Gent, Christopher Jefferson, Ian Miguel, and Peter Nightingale. Breaking conditional symmetry in automated constraint modelling with Conjure. In Proceedings of the 21st European Conference on Artificial Intelligence (ECAI), pages 3-8, 2014. Google Scholar
  6. Ozgur Akgun, Ian Miguel, Christopher Jefferson, Alan M Frisch, and Brahim Hnich. Extensible Automated Constraint Modelling. In Wolfram Burgard and Dan Roth, editors, AAAI 2011 - Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, California, USA, August 7-11, 2011. AAAI Press, 2011. Google Scholar
  7. Roberto Amadini, Maurizio Gabbrielli, and Jacopo Mauro. An enhanced features extractor for a portfolio of constraint solvers. In Proceedings of the 29th annual ACM symposium on applied computing, pages 1357-1359, 2014. Google Scholar
  8. Roberto Amadini, Maurizio Gabbrielli, and Jacopo Mauro. SUNNY-CP: a sequential CP portfolio solver. In Proceedings of the 30th Annual ACM Symposium on Applied Computing, pages 1861-1867, 2015. Google Scholar
  9. Clark Barrett and Cesare Tinelli. Satisfiability modulo theories. In Handbook of model checking, pages 305-343. Springer, 2018. Google Scholar
  10. Thomas Bartz-Beielstein, Carola Doerr, Daan van den Berg, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, William La Cava, Manuel Lopez-Ibanez, et al. Benchmarking in optimization: Best practice and open issues. arXiv preprint arXiv:2007.03488, 2020. Google Scholar
  11. Vahid Beiranvand, Warren Hare, and Yves Lucet. Best practices for comparing optimization algorithms. Optimization and Engineering, 18(4):815-848, 2017. Google Scholar
  12. Armin Biere, Mathias Fleury, and Maximilian Heisinger. CaDiCaL, Kissat, Paracooba entering the SAT competition 2021. In T Balyo, N Froleyks, M Heule, M Iser, M Järvisalo, and M Suda, editors, Proceedings of SAT Competition 2021: Solver and Benchmark Descriptions. Department of Computer Science Report Series B, vol. B-2021-1, Department of Computer Science, University of Helsinki, Helsinki, 2021. Google Scholar
  13. Armin Biere, Marijn Heule, and Hans van Maaren. Handbook of satisfiability, volume 185. IOS press, 2009. Google Scholar
  14. Jakob Bossek and Markus Wagner. Generating instances with performance differences for more than just two algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 1423-1432, 2021. Google Scholar
  15. Geoffrey Chu, Peter J. Stuckey, Andreas Schutt, Thorsten Ehlers, Graeme Gange, and Kathryn Francis. Chuffed, 2018. Available from URL: https://github.com/chuffed/chuffed/.
  16. Arnaud De Coster, Nysret Musliu, Andrea Schaerf, Johannes Schoisswohl, and Kate Smith-Miles. Algorithm selection and instance space analysis for curriculum-based course timetabling. Journal of Scheduling, pages 1-24, 2021. Google Scholar
  17. Luiz Henrique dos Santos Fernandes, Ana Carolina Lorena, and Kate Smith-Miles. Towards understanding clustering problems and algorithms: an instance space analysis. Algorithms, 14(3):95, 2021. Google Scholar
  18. Nils Froleyks, Marijn Heule, Markus Iser, Matti Järvisalo, and Martin Suda. SAT competition 2020. Artificial Intelligence, 301:103572, 2021. Google Scholar
  19. Nils Froleyks, Marijn Heule, Markus Iser, Matti Järvisalo, and Martin Suda. SAT competition 2020. Artificial Intelligence, 301:103572, 2021. URL: https://doi.org/10.1016/j.artint.2021.103572.
  20. Ian P. Gent, Christopher Jefferson, and Ian Miguel. Minion: A fast scalable constraint solver. In Proceedings ECAI 2006, pages 98-102, 2006. Google Scholar
  21. Jesús Giráldez-Cru and Jordi Levy. A modularity-based random SAT instances generator. In Qiang Yang and Michael J. Wooldridge, editors, Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI, pages 1952-1958. AAAI Press, 2015. URL: http://ijcai.org/Abstract/15/277.
  22. Vinasétan Ratheil Houndji, Pierre Schaus, Laurence Wolsey, and Yves Deville. The stockingcost constraint. In International conference on principles and practice of constraint programming, pages 382-397. Springer, 2014. Google Scholar
  23. Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization - 5th International Conference, LION 5, Rome, Italy, volume 6683 of Lecture Notes in Computer Science, pages 507-523. Springer, 2011. URL: https://doi.org/10.1007/978-3-642-25566-3_40.
  24. Lucas Kletzander, Nysret Musliu, and Kate Smith-Miles. Instance space analysis for a personnel scheduling problem. Annals of Mathematics and Artificial Intelligence, 89(7):617-637, 2021. Google Scholar
  25. Stefan Kreter, Andreas Schutt, Peter J Stuckey, and Jürgen Zimmermann. Mixed-integer linear programming and constraint programming formulations for solving resource availability cost problems. European Journal of Operational Research, 266(2):472-486, 2018. Google Scholar
  26. Edward Lam, Peter J Stuckey, Sven Koenig, and TK Kumar. Exact approaches to the multi-agent collective construction problem. In International Conference on Principles and Practice of Constraint Programming, pages 743-758. Springer, 2020. Google Scholar
  27. Kelvin Liu, Kate Smith-Miles, and Alysson Costa. Using instance space analysis to study the bin packing problem, 2020. Google Scholar
  28. Tong Liu, Roberto Amadini, Maurizio Gabbrielli, and Jacopo Mauro. sunny-as2: Enhancing SUNNY for algorithm selection. Journal of Artificial Intelligence Research, 72:329-376, 2021. Google Scholar
  29. Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Mauro Birattari, and Thomas Stützle. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43-58, 2016. Google Scholar
  30. Oden Maron and Andrew W Moore. The racing algorithm: Model selection for lazy learners. Artificial Intelligence Review, 11(1):193-225, 1997. Google Scholar
  31. Michael Marte. Yuck, 2021. Available from https://github.com/informarte/yuck. Google Scholar
  32. Mario A Muñoz, Laura Villanova, Davaatseren Baatar, and Kate Smith-Miles. Instance spaces for machine learning classification. Machine Learning, 107(1):109-147, 2018. Google Scholar
  33. Mario Andrés Muñoz, Tao Yan, Matheus R Leal, Kate Smith-Miles, Ana Carolina Lorena, Gisele L Pappa, and Rômulo Madureira Rodrigues. An instance space analysis of regression problems. ACM Transactions on Knowledge Discovery from Data (TKDD), 15(2):1-25, 2021. Google Scholar
  34. Nicholas Nethercote, Peter J Stuckey, Ralph Becket, Sebastian Brand, Gregory J Duck, and Guido Tack. Minizinc: Towards a standard CP modelling language. In International Conference on Principles and Practice of Constraint Programming, pages 529-543. Springer, 2007. Google Scholar
  35. Peter Nightingale, Özgür Akgün, Ian P Gent, Christopher Jefferson, Ian Miguel, and Patrick Spracklen. Automatically improving constraint models in Savile Row. Artificial Intelligence, 251:35-61, 2017. Google Scholar
  36. Eoin O’Mahony, Emmanuel Hebrard, Alan Holland, Conor Nugent, and Barry O’Sullivan. Using case-based reasoning in an algorithm portfolio for constraint solving. In Irish conference on artificial intelligence and cognitive science, pages 210-216, 2008. Google Scholar
  37. David Pisinger. Where are the hard knapsack problems? Computers & Operations Research, 32(9):2271-2284, 2005. Google Scholar
  38. Olivier Roussel. Controlling a solver execution with the runsolver tool. Journal on Satisfiability, Boolean Modeling and Computation, 7(4):139-144, 2011. Google Scholar
  39. Marius Schneider and Holger H Hoos. Quantifying homogeneity of instance sets for algorithm configuration. In International Conference on Learning and Intelligent Optimization, pages 190-204. Springer, 2012. Google Scholar
  40. Andreas Schutt, Peter J Stuckey, and Andrew R Verden. Optimal carpet cutting. In International Conference on Principles and Practice of Constraint Programming, pages 69-84. Springer, 2011. Google Scholar
  41. Bart Selman, David G. Mitchell, and Hector J. Levesque. Generating hard satisfiability problems. Artificial Intelligence, 81(1-2):17-29, 1996. URL: https://doi.org/10.1016/0004-3702(95)00045-3.
  42. Kate Smith-Miles, Jeffrey Christiansen, and Mario Andrés Muñoz. Revisiting where are the hard knapsack problems? via instance space analysis. Computers & Operations Research, 128:105184, 2021. Google Scholar
  43. Kate Smith-Miles and Jano van Hemert. Discovering the suitability of optimisation algorithms by learning from evolved instances. Annals of Mathematics and Artificial Intelligence, 61(2):87-104, 2011. Google Scholar
  44. Peter J Stuckey, Ralph Becket, and Julien Fischer. Philosophy of the MiniZinc challenge. Constraints, 15(3):307-316, 2010. Google Scholar
  45. Alvaro Torralba, Jendrik Seipp, and Silvan Sievers. Automatic instance generation for classical planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 31, pages 376-384, 2021. Google Scholar
  46. Hafiz Ullah and Sultana Parveen. A literature review on inventory lot sizing problems. Global Journal of Research In Engineering, 10(5), 2010. Google Scholar
  47. Mauro Vallati, Lukás Chrpa, and Thomas Leo McCluskey. What you always wanted to know about the deterministic part of the international planning competition (IPC) 2014 (but were too afraid to ask). Knowledge Engineering Review, 33:e3, 2018. URL: https://doi.org/10.1017/S0269888918000012.
  48. Mario Vanhoucke and Broos Maenhout. On the characterization and generation of nurse scheduling problem instances. European Journal of Operational Research, 196(2):457-467, 2009. Google Scholar
  49. Lin Xu, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. SATzilla: portfolio-based algorithm selection for sat. Journal of artificial intelligence research, 32:565-606, 2008. Google Scholar
  50. Lin Xu, Frank Hutter, Jonathan Shen, Holger H Hoos, and Kevin Leyton-Brown. SATzilla2012: Improved algorithm selection based on cost-sensitive classification models. Proceedings of SAT Challenge, 2012, 2012. Google Scholar
  51. Neng-Fa Zhou and Håkan Kjellerstrand. Optimizing SAT encodings for arithmetic constraints. In International Conference on Principles and Practice of Constraint Programming, pages 671-686. Springer, 2017. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact