SAT-Based Approach for Learning Optimal Decision Trees with Non-Binary Features

Authors Pouya Shati, Eldan Cohen, Sheila McIlraith



PDF
Thumbnail PDF

File

LIPIcs.CP.2021.50.pdf
  • Filesize: 0.66 MB
  • 16 pages

Document Identifiers

Author Details

Pouya Shati
  • Department of Computer Science, University of Toronto, Canada
Eldan Cohen
  • Department of Mechanical and Industrial Engineering, University of Toronto, Canada
Sheila McIlraith
  • Department of Computer Science, University of Toronto, Canada
  • Vector Institute, Toronto, Canada

Cite AsGet BibTex

Pouya Shati, Eldan Cohen, and Sheila McIlraith. SAT-Based Approach for Learning Optimal Decision Trees with Non-Binary Features. In 27th International Conference on Principles and Practice of Constraint Programming (CP 2021). Leibniz International Proceedings in Informatics (LIPIcs), Volume 210, pp. 50:1-50:16, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2021)
https://doi.org/10.4230/LIPIcs.CP.2021.50

Abstract

Decision trees are a popular classification model in machine learning due to their interpretability and performance. Traditionally, decision-tree classifiers are constructed using greedy heuristic algorithms, however these algorithms do not provide guarantees on the quality of the resultant trees. Instead, a recent line of work has studied the use of exact optimization approaches for constructing optimal decision trees. Most of the recent approaches that employ exact optimization are designed for datasets with binary features. While numeric and categorical features can be transformed to binary features, this transformation can introduce a large number of binary features and may not be efficient in practice. In this work, we present a novel SAT-based encoding for decision trees that supports non-binary features and demonstrate how it can be used to solve two well-studied variants of the optimal decision tree problem. We perform an extensive empirical analysis that shows our approach obtains superior performance and is often an order of magnitude faster than the current state-of-the-art exact techniques on non-binary datasets.

Subject Classification

ACM Subject Classification
  • Mathematics of computing → Combinatorial optimization
  • Computing methodologies → Machine learning
Keywords
  • Decision Tree
  • Classification
  • Numeric Data
  • Categorical Data
  • SAT
  • MaxSAT

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads

References

  1. Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. Learning optimal and fair decision trees for non-discriminative decision-making. In AAAI Conference on Artificial Intelligence (AAAI), pages 1418-1426, 2019. Google Scholar
  2. Gaël Aglin, Siegfried Nijssen, and Pierre Schaus. Learning optimal decision trees using caching branch-and-bound search. In AAAI Conference on Artificial Intelligence (AAAI), pages 3146-3153, 2020. Google Scholar
  3. Florent Avellaneda. Efficient inference of optimal decision trees. In AAAI Conference on Artificial Intelligence (AAAI), pages 3195-3202, 2020. Google Scholar
  4. Kristin P Bennett. Global tree optimization: A non-greedy decision tree algorithm. Journal of Computing Science and Statistics, 26:156-160, 1994. Google Scholar
  5. Jeremias Berg, Emir Demirović, and Peter J Stuckey. Core-boosted linear search for incomplete MaxSAT. In International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research (CPAIOR), pages 39-56. Springer, 2019. Google Scholar
  6. Dimitris Bertsimas and Jack Dunn. Optimal classification trees. Machine Learning, 106(7):1039-1082, 2017. Google Scholar
  7. Christian Bessiere, Emmanuel Hebrard, and Barry O’Sullivan. Minimising decision tree size as combinatorial optimisation. In International Conference on Principles and Practice of Constraint Programming (CP), pages 173-187. Springer, 2009. Google Scholar
  8. Armin Biere, Marijn Heule, and Hans van Maaren. Handbook of satisfiability, volume 185. IOS press, 2009. Google Scholar
  9. Leo Breiman, Jerome H Friedman, Richard A Olshen, and Charles J Stone. Classification and regression trees. Wadsworth & Brooks/Cole Advanced Books & Software, 1984. Google Scholar
  10. Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL: http://archive.ics.uci.edu/ml.
  11. Niklas Eén and Niklas Sörensson. Minisat SAT solver, 2003. URL: http://minisat.se/Main.html.
  12. Zhaohui Fu and Sharad Malik. On solving the partial MAX-SAT problem. In International Conference on Theory and Applications of Satisfiability Testing (SAT), pages 252-265. Springer, 2006. Google Scholar
  13. Oktay Günlük, Jayant Kalagnanam, Minhan Li, Matt Menickelly, and Katya Scheinberg. Optimal decision trees for categorical data via integer programming. Journal of Global Optimization, pages 1-28, 2021. Google Scholar
  14. Thomas Hancock, Tao Jiang, Ming Li, and John Tromp. Lower bounds on learning decision lists and trees. Information and Computation, 126(2):114-122, 1996. Google Scholar
  15. Hao Hu, Mohamed Siala, Emmanuel Hébrard, and Marie-José Huguet. Learning optimal decision trees with MaxSAT and its integration in AdaBoost. In International Joint Conference on Artificial Intelligence and Pacific Rim International Conference on Artificial Intelligence (IJCAI-PRICAI), 2020. Google Scholar
  16. Alexey Ignatiev, Joao Marques-Silva, Nina Narodytska, and Peter J Stuckey. Reasoning-based learning of interpretable ML models. In International Joint Conference on Artificial Intelligence (IJCAI), page in press, 2021. Google Scholar
  17. Hyafil Laurent and Ronald L Rivest. Constructing optimal binary decision trees is np-complete. Information processing letters, 5(1):15-17, 1976. Google Scholar
  18. Nina Narodytska, Alexey Ignatiev, Filipe Pereira, Joao Marques-Silva, and IS RAS. Learning optimal decision trees with SAT. In International Joint Conference on Artificial Intelligence (IJCAI), pages 1362-1368, 2018. Google Scholar
  19. Siegfried Nijssen and Elisa Fromont. Optimal constraint-based decision tree induction from itemset lattices. Data Mining and Knowledge Discovery, 21(1):9-51, 2010. Google Scholar
  20. OscaR Team. OscaR: Scala in OR, 2012. URL: https://bitbucket.org/oscarlib/oscar.
  21. J Ross Quinlan. Induction of decision trees. Machine learning, 1(1):81-106, 1986. Google Scholar
  22. J Ross Quinlan. C4. 5: programs for machine learning. Elsevier, 2014. Google Scholar
  23. Hélene Verhaeghe, Siegfried Nijssen, Gilles Pesant, Claude-Guy Quimper, and Pierre Schaus. Learning optimal decision trees using constraint programming. Constraints, 25(3):226-250, 2020. Google Scholar
  24. Sicco Verwer and Yingqian Zhang. Learning optimal classification trees using a binary linear program formulation. In AAAI Conference on Artificial Intelligence (AAAI), pages 1625-1632, 2019. Google Scholar
  25. Jinqiang Yu, Alexey Ignatiev, Peter J Stuckey, and Pierre Le Bodic. Computing optimal decision sets with sat. In International Conference on Principles and Practice of Constraint Programming (CP), pages 952-970. Springer, 2020. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail