Document Open Access Logo

Combining Constraint Programming Reasoning with Large Language Model Predictions

Authors Florian Régin, Elisabetta De Maria, Alexandre Bonlarron

PDF
Thumbnail PDF

File

LIPIcs.CP.2024.25.pdf
  • Filesize: 0.84 MB
  • 18 pages

Document Identifiers

Author Details

Florian Régin
  • Université Côte d'Azur, I3S, CNRS, Sophia Antipolis, France
Elisabetta De Maria
  • Université Côte d'Azur, I3S, CNRS, Sophia Antipolis, France
Alexandre Bonlarron
  • Université Côte d'Azur, Inria, Sophia Antipolis, France
  • Université Côte d'Azur, I3S, CNRS, Sophia Antipolis, France

Acknowledgements

We thank Jack Massey for his help in reproducing the benchmarks used as baseline in Section 4.1.1.

Cite AsGet BibTex

Florian Régin, Elisabetta De Maria, and Alexandre Bonlarron. Combining Constraint Programming Reasoning with Large Language Model Predictions. In 30th International Conference on Principles and Practice of Constraint Programming (CP 2024). Leibniz International Proceedings in Informatics (LIPIcs), Volume 307, pp. 25:1-25:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024)
https://doi.org/10.4230/LIPIcs.CP.2024.25

Abstract

Constraint Programming (CP) and Machine Learning (ML) face challenges in text generation due to CP’s struggle with implementing "meaning" and ML’s difficulty with structural constraints. This paper proposes a solution by combining both approaches and embedding a Large Language Model (LLM) in CP. The LLM handles word generation and meaning, while CP manages structural constraints. This approach builds on GenCP, an improved version of On-the-fly Constraint Programming Search (OTFS) using LLM-generated domains. Compared to Beam Search (BS), a standard NLP method, this combined approach (GenCP with LLM) is faster and produces better results, ensuring all constraints are satisfied. This fusion of CP and ML presents new possibilities for enhancing text generation under constraints.

Subject Classification

ACM Subject Classification
  • Theory of computation → Constraint and logic programming
Keywords
  • Solver and Tools
  • ML-augmented CP
  • Constrained Text Generation
  • ML alongside CO

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Andrea Bartolini, Michele Lombardi, Michela Milano, and Luca Benini. Neuron constraints to model complex real-world problems. In Principles and Practice of Constraint Programming-CP 2011: 17th International Conference, CP 2011, Perugia, Italy, September 12-16, 2011, Proceedings, volume 6876, page 115. Springer, 2011. Google Scholar
  2. Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: A methodological tour d’horizon. European Journal of Operational Research, 290(2):405-421, 2021. URL: https://doi.org/10.1016/j.ejor.2020.07.063.
  3. Christian Bessière. Arc-consistency in dynamic constraint satisfaction problems. In Thomas L. Dean and Kathleen R. McKeown, editors, Proceedings of the 9th National Conference on Artificial Intelligence, Anaheim, CA, USA, July 14-19, 1991, Volume 1, pages 221-226. AAAI Press / The MIT Press, 1991. URL: http://www.aaai.org/Library/AAAI/1991/aaai91-035.php.
  4. Alexandre Bonlarron, Aurélie Calabrèse, Pierre Kornprobst, and Jean-Charles Régin. Constraints first: a new mdd-based model to generate sentences under constraints. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 1893-1901, 2023. Google Scholar
  5. Alexandre Bonlarron and Jean-Charles Régin. Intertwining cp and nlp: The generation of unreasonably constrained sentences. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, 2024. To appear. Google Scholar
  6. Alexandre Bonlarron and Jean-Charles Régin. Markov constraint as large language model surrogate. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, 2024. To appear. Google Scholar
  7. Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, Jennifer C. Lai, and Robert L. Mercer. An estimate of an upper bound for the entropy of English. Computational Linguistics, 18(1):31-40, 1992. URL: https://aclanthology.org/J92-1002.
  8. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. Google Scholar
  9. Quentin Cappart, Thierry Moisan, Louis-Martin Rousseau, Isabeau Prémont-Schwarz, and Andre A. Cire. Combining reinforcement learning and constraint programming for combinatorial optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 35(5):3677-3687, May 2021. URL: https://doi.org/10.1609/aaai.v35i5.16484.
  10. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. URL: https://arxiv.org/abs/2403.04132.
  11. Parag Pravin Dakle, Serdar Kadıoğlu, Karthik Uppuluri, Regina Politi, Preethi Raghavan, SaiKrishna Rallabandi, and Ravisutha Srinivasamurthy. Ner4opt: Named entity recognition for ;optimization modelling from ;natural language. In Integration of Constraint Programming, Artificial Intelligence, and Operations Research: 20th International Conference, CPAIOR 2023, Nice, France, May 29 –June 1, 2023, Proceedings, pages 299-319, Berlin, Heidelberg, 2023. Springer-Verlag. URL: https://doi.org/10.1007/978-3-031-33271-5_20.
  12. Rina Dechter and Avi Dechter. Belief maintenance in dynamic constraint networks. In Howard E. Shrobe, Tom M. Mitchell, and Reid G. Smith, editors, Proceedings of the 7th National Conference on Artificial Intelligence, St. Paul, MN, USA, August 21-26, 1988, pages 37-42. AAAI Press / The MIT Press, 1988. URL: http://www.aaai.org/Library/AAAI/1988/aaai88-007.php.
  13. Eugene C. Freuder. In pursuit of the holy grail. Constraints, 2(1):57-61, 1997. URL: https://doi.org/10.1023/A:1009749006768.
  14. Eugene C. Freuder. Conversational modeling for constraint satisfaction. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20):22592-22597, March 2024. URL: https://doi.org/10.1609/aaai.v38i20.30268.
  15. Cristina Garbacea and Qiaozhu Mei. Why is constrained neural language generation particularly challenging? arXiv preprint arXiv:2206.05395, 2022. Google Scholar
  16. Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W. Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. CoRR, abs/2103.13630, 2021. URL: https://arxiv.org/abs/2103.13630.
  17. Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535-1546, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/P17-1141.
  18. Nan Jiang, Maosen Zhang, Willem-Jan Van Hoeve, and Yexiang Xue. Constraint reasoning embedded structured prediction. J. Mach. Learn. Res., 23(1), January 2022. Google Scholar
  19. U Junker, F Rossi, P van Beek, and T Walsh. Handbook of constraint programming. Chapter Configuration, 2006. Google Scholar
  20. Dan Jurafsky and James H. Martin. Speech and language processing : an introduction to natural language processing, computational linguistics, and speech recognition. Pearson Prentice Hall, Upper Saddle River, N.J., 2009. URL: http://www.amazon.com/Speech-Language-Processing-2nd-Edition/dp/0131873210/ref=pd_bxgy_b_img_y.
  21. Elias Khalil, Pierre Le Bodic, Le Song, George Nemhauser, and Bistra Dilkina. Learning to branch in mixed integer programming. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), February 2016. URL: https://doi.org/10.1609/aaai.v30i1.10080.
  22. Elias B. Khalil, Bistra Dilkina, George L. Nemhauser, Shabbir Ahmed, and Yufen Shao. Learning to run heuristics in tree search. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 659-666, 2017. URL: https://doi.org/10.24963/ijcai.2017/92.
  23. James Kotary, Ferdinando Fioretto, Pascal van Hentenryck, and Bryan Wilder. End-to-end constrained optimization learning: A survey. In 30th International Joint Conference on Artificial Intelligence, IJCAI 2021, pages 4475-4482. International Joint Conferences on Artificial Intelligence, 2021. Google Scholar
  24. Connor Lawless, Jakob Schoeffer, Lindy Le, Kael Rowan, Shilad Sen, Cristina St. Hill, Jina Suh, and Bahareh Sarrafzadeh. "i want it that way": Enabling interactive decision support using large language models and constraint programming, 2024. URL: https://arxiv.org/abs/2312.06908.
  25. Yixian Liu, Liwen Zhang, Wenjuan Han, Yue Zhang, and Kewei Tu. Constrained text generation with global guidance - case study on commongen. CoRR, abs/2103.07170, 2021. URL: https://arxiv.org/abs/2103.07170.
  26. Michele Lombardi, Michela Milano, and Andrea Bartolini. Empirical decision model learning. Artificial Intelligence, 244:343-367, 2017. Combining Constraint Solving with Mining and Learning. URL: https://doi.org/10.1016/j.artint.2016.01.005.
  27. Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. NeuroLogic A*esque decoding: Constrained text generation with lookahead heuristics. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 780-799, Seattle, United States, July 2022. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/2022.naacl-main.57.
  28. Mohsen Nafar and Michael Römer. Using clustering to strengthen decision diagram bounds for discrete optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 38(8):8082-8089, March 2024. URL: https://doi.org/10.1609/aaai.v38i8.28647.
  29. François Pachet and Pierre Roy. Markov constraints: Steerable generation of markov sequences. Constraints, 16(2):148-172, April 2011. URL: https://doi.org/10.1007/s10601-010-9101-4.
  30. Alexandre Papadopoulos, Pierre Roy, Jean-Charles Régin, and François Pachet. Generating all possible palindromes from ngram corpora. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, pages 2489-2495. AAAI Press, 2015. Google Scholar
  31. Guillaume Perez and Jean-Charles Régin. MDDs: Sampling and probability constraints. In Proceedings of the International Conference on Principles and Practice of Constraint Programming, pages 226-242, 2017. URL: https://doi.org/10.1007/978-3-319-66158-2_15.
  32. Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1314-1324, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/N18-1119.
  33. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021. URL: https://arxiv.org/abs/2112.10752.
  34. Florian Régin and Elisabetta De Maria. Using on-the-fly model checking to improve constraint programming for dynamic problems. In 2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI), pages 393-398, 2023. URL: https://doi.org/10.1109/ICTAI59109.2023.00063.
  35. Jialin Song, ravi lanka, Yisong Yue, and Bistra Dilkina. A general large neighborhood search framework for solving integer linear programs. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20012-20023. Curran Associates, Inc., 2020. URL: https://proceedings.neurips.cc/paper_files/paper/2020/file/e769e03a9d329b2e864b4bf4ff54ff39-Paper.pdf.
  36. Damien Sprockeels and Peter Van Roy. Expressing musical ideas with constraint programming using a model of tonal harmony. In International Joint Conference on Artificial Intelligence, 2024. To appear. Google Scholar
  37. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. cite arxiv:2302.13971. URL: http://arxiv.org/abs/2302.13971.
  38. Dimos Tsouros, Hélène Verhaeghe, Serdar Kadıoğlu, and Tias Guns. Holy grail 2.0: From natural language to constraint models, 2023. URL: https://arxiv.org/abs/2308.01589.
  39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017. Google Scholar
  40. Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik R Narasimhan. COLLIE: Systematic construction of constrained text generation tasks. In The Twelfth International Conference on Learning Representations, 2024. URL: https://openreview.net/forum?id=kxgSlyirUZ.
  41. Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, Ziyang Ma, Liumeng Xue, Ziyu Wang, Qin Liu, Tianyu Zheng, Yizhi Li, Yinghao Ma, Yiming Liang, Xiaowei Chi, Ruibo Liu, Zili Wang, Pengfei Li, Jingcheng Wu, Chenghua Lin, Qifeng Liu, Tao Jiang, Wenhao Huang, Wenhu Chen, Emmanouil Benetos, Jie Fu, Gus Xia, Roger Dannenberg, Wei Xue, Shiyin Kang, and Yike Guo. Chatmusician: Understanding and generating music intrinsically with llm, 2024. URL: https://arxiv.org/abs/2402.16153.
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact