Document Open Access Logo

Transformer-Based Feature Learning for Algorithm Selection in Combinatorial Optimisation

Authors Alessio Pellegrino , Özgür Akgün , Nguyen Dang , Zeynep Kiziltan , Ian Miguel

PDF HTML 5

Files

Thumbnail PDF
PDF
LIPIcs.CP.2025.31.pdf
  • Filesize: 1.29 MB
  • 22 pages
HTML5 Logo HTML (experimental)
LIPIcs.CP.2025.31.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Artificial intelligence
Keywords
  • Constraint modelling
  • algorithm selection
  • feature extraction
  • machine learning
  • transformer architecture

Metrics

Abstract

Given a combinatorial optimisation problem, there are typically multiple ways of modelling it for presentation to an automated solver. Choosing the right combination of model and target solver can have a significant impact on the effectiveness of the solving process. The best combination of model and solver can also be instance-dependent: there may not exist a single combination that works best for all instances of the same problem. We consider the task of building machine learning models to automatically select the best combination for a problem instance. Critical to the learning process is to define instance features, which serve as input to the selection model. Our contribution is the automatic learning of instance features directly from the high-level representation of a problem instance using a transformer encoder. We evaluate the performance of our approach using the Essence modelling language via a case study of three problem classes.

Cite As Get BibTex

Alessio Pellegrino, Özgür Akgün, Nguyen Dang, Zeynep Kiziltan, and Ian Miguel. Transformer-Based Feature Learning for Algorithm Selection in Combinatorial Optimisation. In 31st International Conference on Principles and Practice of Constraint Programming (CP 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 340, pp. 31:1-31:22, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.CP.2025.31

Author Details

Alessio Pellegrino
  • Department of Mathematics and Computer Science, University of Southern Denmark, Denmark
Özgür Akgün
  • School of Computer Science, University of St Andrews, Scotland
Nguyen Dang
  • School of Computer Science, University of St Andrews, Scotland
Zeynep Kiziltan
  • Dept. of Computer Science and Engineering, University of Bologna, Italy
Ian Miguel
  • School of Computer Science, University of St Andrews, Scotland

Funding

This work was supported by the European Union’s Justice programme, under GA No 101087342, POLINE (Principles Of Law In National and European VAT), by a scholarship from the Department of Computer Science and Engineering of the University of Bologna, and by UK EPSRC grant EP/V027182/1.

Supplementary Materials

References

  1. M. Ahmed, R. Seraj, and S. M. S. Islam. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9(8):1295, 2020. Google Scholar
  2. Ö. Akgün, A. M Frisch, I. P Gent, C. Jefferson, I. Miguel, and P. Nightingale. Conjure: Automatic generation of constraint models from problem specifications. Artificial Intelligence, 310:103751, 2022. URL: https://doi.org/10.1016/J.ARTINT.2022.103751.
  3. M. Alissa, K. Sim, and E. Hart. Automated algorithm selection: from feature-based to feature-free approaches. Journal of Heuristics, 29(1):1-38, 2023. URL: https://doi.org/10.1007/S10732-022-09505-4.
  4. R. Amadini, M. Gabbrielli, and J. Mauro. An enhanced features extractor for a portfolio of constraint solvers. In Proceedings of the 29th annual ACM symposium on applied computing, pages 1357-1359, 2014. Google Scholar
  5. R. Amadini, M. Gabbrielli, and J. Mauro. SUNNY: a lazy portfolio approach for constraint solving. Theory and Practice of Logic Programming, 14(4-5):509-524, 2014. URL: https://doi.org/10.1017/S1471068414000179.
  6. R. Amadini, M. Gabbrielli, and J. Mauro. SUNNY-CP: a sequential CP portfolio solver. In Proceedings of the 30th Annual ACM Symposium on Applied Computing, pages 1861-1867, 2015. Google Scholar
  7. J. Ansel, E. Yang, H. He, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski, et al. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 929-947, 2024. Google Scholar
  8. C. Ansótegui, M. Sellmann, and K. Tierney. Self-configuring cost-sensitive hierarchical clustering with recourse. In Proceedings of the 24th International Conference on Principles and Practice of Constraint Programming, pages 524-534. Springer, 2018. Google Scholar
  9. I. Beltagy, M. E P.s, and A. Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. Google Scholar
  10. A. Biere and M. Fleury. Gimsatul, IsaSAT and Kissat entering the SAT Competition 2022. In Tomas Balyo, Marijn Heule, Markus Iser, Matti Järvisalo, and Martin Suda, editors, Proceedings of SAT Competition 2022 - Solver and Benchmark Descriptions, volume B-2022-1 of Department of Computer Science Series of Publications B, pages 10-11. University of Helsinki, 2022. Google Scholar
  11. B. Bischl, P. Kerschke, L. Kotthoff, M. Lindauer, Y. Malitsky, A. Fréchette, H. Hoos, F. Hutter, K. Leyton-Brown, K. Tierney, and J. Vanschoren. ASlib: A benchmark library for algorithm selection. Artificial Intelligence, 237:41-58, 2016. URL: https://doi.org/10.1016/j.artint.2016.04.003.
  12. D. Bridge, E. O’Mahony, and B. O’Sullivan. Case-based reasoning for autonomous constraint solving. In Autonomous search, pages 73-95. Springer, 2012. URL: https://doi.org/10.1007/978-3-642-21434-9_4.
  13. G. Cenikj, G. Petelin, and T. Eftimov. Transoptas: Transformer-based algorithm selection for single-objective optimization. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 403-406, 2024. Google Scholar
  14. T. Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016. Google Scholar
  15. Chuffed Developers. Chuffed, a lazy clause generation solver. https://github.com/chuffed/chuffed. Accessed: 2024-07-05.
  16. Conjure developers. Essence catalog: A collection of problem specifications in essence. https://github.com/conjure-cp/EssenceCatalog, 2024.
  17. A. Crespo Márquez. The curse of dimensionality. In Digital Maintenance Management: Guiding Digital Transformation in Maintenance, pages 67-86. Springer, 2022. Google Scholar
  18. N. Dang. A portfolio-based analysis method for competition results. arXiv preprint arXiv:2205.15414, 2022. Google Scholar
  19. N. Dang, Özgür Akgün, J. Espasa, I. Miguel, and P. Nightingale. A Framework for Generating Informative Benchmark Instances. In Christine Solnon, editor, Proceedings of the 28th International Conference on Principles and Practice of Constraint Programming, volume 235 of Leibniz International Proceedings in Informatics (LIPIcs), pages 18:1-18:18, Dagstuhl, Germany, 2022. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. URL: https://doi.org/10.4230/LIPIcs.CP.2022.18.
  20. J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. Google Scholar
  21. F. Didier, L. Perron, S. Mohajeri, S. A. Gay, T. Cuvelier, and V. Furnon. OR-Tools' vehicle routing solver: a generic constraint-programming solver with heuristic search for routing problems, 2023. Google Scholar
  22. M. Dincbas, H. Simonis, and P. Van Hentenryck. Solving the car-sequencing problem in constraint logic programming. In Proceedings of the European Conference on Artificial Intelligence, pages 290-295, 1988. Google Scholar
  23. A. M. Frisch, W. Harvey, C. Jefferson, B. Martínez-Hernández, and I. Miguel. Essence: A constraint language for specifying combinatorial problems. Constraints, 13(3):268-306, 2008. URL: https://doi.org/10.1007/s10601-008-9047-y.
  24. I. P Gent. Two results on car-sequencing problems. Report University of Strathclyde, APES-02-98, 7, 1998. Google Scholar
  25. Tias Guns. Increasing modeling language convenience with a universal n-dimensional array, cppy as python-embedded example. In Proceedings of the 18th workshop on Constraint Modelling and Reformulation at CP (Modref 2019), volume 19, 2019. Google Scholar
  26. K. Gurney. An introduction to neural networks. CRC press, 2018. Google Scholar
  27. A. Handler. An empirical study of semantic similarity in wordnet and word2vec, 2014. Google Scholar
  28. F. Hutter, Holger H Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Proceedings of the 5th International Conference on Learning and Intelligent Optimization (LION), pages 507-523. Springer, 2011. Google Scholar
  29. IBM. Ilog cplex optimization studio: Cplex optimizer. https://www.ibm.com/products/ilog-cplex-optimization-studio/cplex-optimizer, 2022.
  30. L. Kampel and D. E Simos. A survey on the state of the art of complexity problems for covering arrays. Theoretical Computer Science, 800:107-124, 2019. URL: https://doi.org/10.1016/J.TCS.2019.10.019.
  31. P. Kerschke, H. H Hoos, F. Neumann, and H. Trautmann. Automated algorithm selection: Survey and perspectives. Evolutionary computation, 27(1):3-45, 2019. URL: https://doi.org/10.1162/EVCO_A_00242.
  32. Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. Google Scholar
  33. R. Kora and A. Mohammed. A comprehensive review on transformers models for text classification. In 2023 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), pages 1-7, 2023. URL: https://doi.org/10.1109/MIUCC58832.2023.10278387.
  34. L. Kotthoff. Algorithm selection for combinatorial search problems: A survey. In Data mining and constraint programming: Foundations of a cross-disciplinary approach, pages 149-190. Springer, 2016. URL: https://doi.org/10.1007/978-3-319-50137-6_7.
  35. L. Kotthoff, P. Kerschke, H. Hoos, and H. Trautmann. Improving the state of the art in inexact TSP solving using per-instance algorithm selection. In LION 9, pages 202-217. Springer, 2015. Google Scholar
  36. A. Krizhevsky, I. Sutskever, and G. E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 2012. Google Scholar
  37. M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. B., T. Ruhkopf, R. Sass, and F. Hutter. SMAC3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research, 23(54):1-9, 2022. URL: http://jmlr.org/papers/v23/21-0888.html.
  38. M. Lindauer, H. H Hoos, F. H., and T. Schaub. Autofolio: An automatically configured algorithm selector. Journal of Artificial Intelligence Research, 53:745-778, 2015. URL: https://doi.org/10.1613/JAIR.4726.
  39. T. Liu, R. Amadini, M. Gabbrielli, and J. Mauro. sunny-as2: Enhancing SUNNY for algorithm selection. Journal of Artificial Intelligence Research, 72:329-376, 2021. URL: https://doi.org/10.1613/JAIR.1.13116.
  40. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. URL: https://arxiv.org/abs/1907.11692.
  41. N. Nethercote, P. J Stuckey, R. Becket, S. Brand, G. J Duck, and G. Tack. MiniZinc: Towards a standard CP modelling language. In Proceedings of the 13th International Conference on Principles and Practice of Constraint Programming, pages 529-543. Springer, 2007. Google Scholar
  42. P. Nightingale, Ö. Akgün, I. P Gent, C. Jefferson, I. Miguel, and Patrick Spracklen. Automatically improving constraint models in Savile Row. Artificial Intelligence, 251:35-61, 2017. URL: https://doi.org/10.1016/J.ARTINT.2017.07.001.
  43. E. Nudelman, K. Leyton-Brown, A. Devkar, Y. Shoham, and H. Hoos. SATzilla: An algorithm portfolio for SAT. Solver description, SAT competition, 2004, 2004. Google Scholar
  44. E. O’Mahony, E. Hebrard, A. Holland, C. Nugent, and B. O’Sullivan. Using case-based reasoning in an algorithm portfolio for constraint solving. In Irish conference on artificial intelligence and cognitive science, pages 210-216, 2008. Google Scholar
  45. Alessio Pellegrino, Özgür Akgün, Nguyen Dang, Zeynep Kiziltan, and Ian Miguel. EFE repository. Software, version 1.0., (visited on 2025-07-22). URL: https://github.com/SeppiaBrilla/EFE_project
    Software Heritage Logo archived version
    full metadata available at: https://doi.org/10.4230/artifacts.24086
  46. R. Qasim, W. H. Bangyal, M. A Alqarni, and A. Ali Almazroi. A fine-tuned bert-based transfer learning approach for text classification. Journal of healthcare engineering, 2022(1):3498123, 2022. Google Scholar
  47. A. Rangamani, M. Lindegaard, T. GA.ti, and T. A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning, pages 28729-28745. PMLR, 2023. Google Scholar
  48. J. R Rice. The algorithm selection problem. In Advances in computers, volume 15, pages 65-118. Elsevier, 1976. URL: https://doi.org/10.1016/S0065-2458(08)60520-3.
  49. M. V. Seiler, J. Rook, J. Heins, O. L. Preuß, J. Bossek, and H. Trautmann. Using reinforcement learning for per-instance algorithm configuration on the TSP. In IEEE Symposium Series on Computational Intelligence, pages 361-368. IEEE, 2023. Google Scholar
  50. H. Shavit and H. H Hoos. Revisiting SATZilla features in 2024. In 27th International Conference on Theory and Applications of Satisfiability Testing (SAT 2024), pages 27-1. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2024. Google Scholar
  51. B. M Smith. Modelling. In Foundations of Artificial Intelligence, volume 2, pages 377-406. Elsevier, 2006. URL: https://doi.org/10.1016/S1574-6526(06)80015-5.
  52. P. Spracklen, N. Dang, Ö. Akgün, and I. Miguel. Automated streamliner portfolios for constraint satisfaction problems. Artificial Intelligence, 319:103915, 2023. URL: https://doi.org/10.1016/J.ARTINT.2023.103915.
  53. S. Sun, W. Chen, L. Wang, X. Liu, and T. Liu. On the depth of deep neural networks: A theoretical view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30(1), March 2016. URL: https://doi.org/10.1609/aaai.v30i1.10243.
  54. M. Triska and N. Musliu. An effective greedy heuristic for the social golfer problem. Annals of Operations Research, 194(1):413-425, 2012. URL: https://doi.org/10.1007/S10479-011-0866-7.
  55. F. Ulrich-Oltean, P. Nightingale, and J. A. Walker. Selecting SAT encodings for pseudo-boolean and linear integer constraints. In Proceedings of the 28th International Conference on Principles and Practice of Constraint Programming. LIPICS, 2022. Google Scholar
  56. F. Ulrich-Oltean, P. Nightingale, and J. A. Walker. Learning to select SAT encodings for pseudo-boolean and linear integer constraints. Constraints, 28(3):397-426, 2023. URL: https://doi.org/10.1007/S10601-023-09364-1.
  57. F. Ulrich-Oltean, P. Nightingale, and J. A. Walker. IndiCon: Selecting SAT encodings for individual pseudo-boolean and linear integer constraints. In 2024 IEEE 36th International Conference on Tools with Artificial Intelligence (ICTAI), pages 36-42. IEEE, 2024. Google Scholar
  58. M. Vallati, L. Chrpa, and D. Kitchin. ASAP: an automatic algorithm selection approach for planning. International Journal on Artificial Intelligence Tools, 23(06):1460032, 2014. URL: https://doi.org/10.1142/S021821301460032X.
  59. A Vaswani. Attention is all you need. Advances in Neural Information Processing, 30, 2017. Google Scholar
  60. L. Xu, F. Hutter, Holger H Hoos, and K. Leyton-Brown. SATzilla: portfolio-based algorithm selection for SAT. Journal of Artificial Intelligence Research, 32:565-606, 2008. URL: https://doi.org/10.1613/JAIR.2490.
  61. L. Xu, F. Hutter, J. Shen, H. H Hoos, and K. Leyton-Brown. SATzilla2012: improved algorithm selection based on cost-sensitive classification models. Proceedings of SAT Challenge, pages 57-58, 2012. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact