Document Open Access Logo

Neurosymbolic Large Neighbourhood Search

Authors Arnaud Delage-Reid , Gilles Pesant , Amal Zouaq

PDF

File

Thumbnail PDF
PDF
LIPIcs.CP.2026.18.pdf
  • Filesize: 2.56 MB
  • 18 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Artificial intelligence
  • Computing methodologies → Machine learning
  • Mathematics of computing → Combinatorial optimization
Keywords
  • Large Neighbourhood Search
  • Neurosymbolic AI
  • Constraint Programming
  • Masked Language Model
  • CPBP Solver

Metrics

Abstract

Recent advances in ai have spurred interest in NeSy architectures that integrate neural and symbolic methods. In particular, combining a Constraint Programming (cp) model with a language model for constrained sequence generation tasks allows the neural component to capture domain knowledge while cp enforces structural constraints. In this paper we propose combining cp with a Masked Language Model (mlm) to perform Large Neighbourhood Search (lns). Unlike conventional left-to-right Large Language Models, mlm s can complete sequences with gaps in arbitrary positions, making them well-suited for this task. Meanwhile, lns provides a cp-based iterative framework to explore constrained subspaces whenever searching the whole space would be intractable. We evaluate NeSylns on tasks in constrained text generation and molecule discovery. Our experiments show that it can quickly generate many high-quality sentences and molecules, even for highly-constrained tasks.

Cite As Get BibTex

Arnaud Delage-Reid, Gilles Pesant, and Amal Zouaq. Neurosymbolic Large Neighbourhood Search. In 32nd International Conference on Principles and Practice of Constraint Programming (CP 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 379, pp. 18:1-18:18, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.CP.2026.18

Author Details

Arnaud Delage-Reid
  • Polytechnique Montréal, Canada
Gilles Pesant
  • Polytechnique Montréal, Canada
  • IVADO, Montréal, Canada
Amal Zouaq
  • Polytechnique Montréal, LAMA-WeST Lab, Canada
  • MILA, Montréal, Canada

Funding

This research was made possible by IVADO grant RG-2025-761206 funded through the Canada First Research Excellence Fund. We also acknowledge the financial support provided by the FRQS (Fonds de Recherche du Québec – Santé) Chair in Artificial Intelligence and Digital Health as well as NSERC Discovery grant RGPIN-2023-05705.

Acknowledgements

This research was enabled in part by support provided by Calcul Québec (https://calculquebec.ca) and the Digital Research Alliance of Canada (https://alliancecan.ca).

Supplementary Materials

References

  1. Mohaddeseh Bastan, Mihai Surdeanu, and Niranjan Balasubramanian. NEUROSTRUCTURAL DECODING: Neural Text Generation with Structural Constraints. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9496-9510, Toronto, Canada, 2023. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/2023.acl-long.528.
  2. Samuel Belkadi, Libo Ren, Nicolo Micheletti, Lifeng Han, and Goran Nenadic. Generating Synthetic Free-text Medical Records with Low Re-identification Risk using Masked Language Modeling. In Abteen Ebrahimi, Samar Haider, Emmy Liu, Sammar Haider, Maria Leonor Pacheco, and Shira Wein, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop), pages 200-206, Albuquerque, USA, 2025. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/2025.naacl-srw.20.
  3. Alexandre Bonlarron, Aurélie Calabrèse, Pierre Kornprobst, and Jean-Charles Régin. Constraints First: A New MDD-based Model to Generate Sentences Under Constraints. In Thirty-Second International Joint Conference on Artificial Intelligence, volume 3, pages 1893-1901, 2023. URL: https://doi.org/10.24963/ijcai.2023/210.
  4. Alexandre Bonlarron, Florian Régin, Elisabetta De Maria, and Jean-Charles Régin. Large language model meets constraint propagation. In James Kwok, editor, Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-25, pages 10036-10044. International Joint Conferences on Artificial Intelligence Organization, August 2025. URL: https://doi.org/10.24963/ijcai.2025/1115.
  5. Alexandre Bonlarron and Jean-Charles Régin. Markov Constraint as Large Language Model Surrogate. In Thirty-Third International Joint Conference on Artificial Intelligence, volume 3, pages 1844-1852, 2024. URL: https://doi.org/10.24963/ijcai.2024/204.
  6. Frédéric Boussemart, Fred Hemery, Christophe Lecoutre, and Lakhdar Sais. Boosting systematic search by weighting constraints. In Ramón López de Mántaras and Lorenza Saitta, editors, Proceedings of the 16th Eureopean Conference on Artificial Intelligence, ECAI'2004, Including Prestigious Applicants of Intelligent Systems, PAIS 2004, Valencia, Spain, August 22-27, 2004, pages 146-150. IOS Press, 2004. Google Scholar
  7. Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(Volume 5):411-444, 2022. URL: https://doi.org/10.1146/annurev-control-042920-020211.
  8. Junyang Cai, Serdar Kadioğlu, and Bistra Dilkina. BALANS: Multi-armed bandits-based adaptive large neighborhood search for mixed-integer programming problems. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, Ijcai '25, Montreal, Canada, 2025. URL: https://doi.org/10.24963/ijcai.2025/286.
  9. Jaymari Chua, Yun Li, Shiyi Yang, Chen Wang, and Lina Yao. AI Safety in Generative AI Large Language Models: A Survey. URL: https://doi.org/10.48550/arXiv.2407.18369.
  10. Paolo Dragone, Stefano Teso, and Andrea Passerini. Neuro-Symbolic Constraint Programming for Structured Prediction. In Artur d'Avila Garcez and Ernesto Jiménez-Ruiz, editors, Proceedings of the 15th International Workshop on Neural-Symbolic Learning and Reasoning, volume 2986 of CEUR Workshop Proceedings, pages 6-14, Virtual conference, October, October 2021. CEUR. URL: https://ceur-ws.org/Vol-2986/paper2.pdf.
  11. Entropy. Entropy/gpt2_zinc_87m ⋅ Hugging Face. URL: https://huggingface.co/entropy/gpt2_zinc_87m.
  12. Yingwen Fu, Wenjie Ou, Zhou Yu, and Yue Lin. Effective Unsupervised Constrained Text Generation based on Perturbed Masking. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 1417-1427, Dublin, Ireland, 2022. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/2022.findings-acl.111.
  13. Taoan Huang, Aaron M. Ferber, Yuandong Tian, Bistra Dilkina, and Benoit Steiner. Searching Large Neighborhoods for Integer Linear Programs with Contrastive Learning. In Proceedings of the 40th International Conference on Machine Learning, pages 13869-13890. PMLR, 2023. URL: https://proceedings.mlr.press/v202/huang23g.html.
  14. F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker. Perplexity - a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63, 2005. URL: https://doi.org/10.1121/1.2016299.
  15. Daniel Jurafsky and James H. Martin. Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition with language models, 3rd edition, 2026. Online manuscript. Google Scholar
  16. Michele Lombardi and Pierre Schaus. Cost Impact Guided LNS. In Helmut Simonis, editor, Integration of AI and OR Techniques in Constraint Programming, volume 8451, pages 293-300. Springer International Publishing, Cham, 2014. URL: https://doi.org/10.1007/978-3-319-07046-9_21.
  17. Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4288-4299, Online, 2021. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/2021.naacl-main.339.
  18. Jean-Baptiste Mairy, Yves Deville, and Pascal Van Hentenryck. Reinforced Adaptive Large Neighborhood Search. In Eightth International Workshop on Local Search Techniques in Constraint Satisfaction (LSCS2011), 2011. Google Scholar
  19. Virasone Manibod, David Saikali, and Gilles Pesant. Constrained Sequential Inference in Machine Learning Using Constraint Programming. In Thirty-Fourth International Joint Conference on Artificial Intelligence, volume 1, pages 2628-2636, 2025. URL: https://doi.org/10.24963/ijcai.2025/293.
  20. J. S. Mansfield, N. Atilgan, A. M. Lewis, and G. E Legge. Extending the MNREAD sentence corpus: Computer-generated sentences for measuring visual performance in reading. Vision research, 158:11-18, 2019. URL: https://doi.org/10.1016/j.visres.2019.01.010.
  21. Merriam-Webster. How many words are there in English? | Merriam-Webster. URL: https://www.merriam-webster.com/help/faq-how-many-english-words.
  22. Meta. Meta-llama/Llama-3.1-8B ⋅ Hugging Face. URL: https://huggingface.co/meta-llama/Llama-3.1-8B.
  23. Nicolo Micheletti, Samuel Belkadi, Lifeng Han, and Goran Nenadic. Exploration of Masked and Causal Language Modelling for Text Generation. URL: https://doi.org/10.48550/arXiv.2405.12630.
  24. Maxime Mulamba, Jayanta Mandi, Ali İrfan Mahmutoğulları, and Tias Guns. Perception-based constraint solving for sudoku images. Constraints, 29(1-2):112-151, 2024. URL: https://doi.org/10.1007/s10601-024-09372-9.
  25. Oxford English Dictionnary. Frequency. URL: https://www.oed.com/information/understanding-entries/frequency/.
  26. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A Method for Automatic Evaluation of Machine Translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. URL: https://doi.org/10.3115/1073083.1073135.
  27. Laurent Perron, Paul Shaw, and Vincent Furnon. Propagation Guided Large Neighborhood Search. In Mark Wallace, editor, Principles and Practice of Constraint Programming - CP 2004, pages 468-481, Berlin, Heidelberg, 2004. Springer. URL: https://doi.org/10.1007/978-3-540-30201-8_35.
  28. Gilles Pesant. From support propagation to belief propagation in constraint programming. Journal of Artificial Intelligence Research, 66:123-150, 2019. URL: https://doi.org/10.1613/JAIR.1.11487.
  29. Florian Régin, Elisabetta De Maria, and Alexandre Bonlarron. Combining Constraint Programming Reasoning with Large Language Model Predictions. In Paul Shaw, editor, 30th International Conference on Principles and Practice of Constraint Programming (CP 2024), volume 307 of Leibniz International Proceedings in Informatics (LIPIcs), pages 25:1-25:18, Dagstuhl, Germany, 2024. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. URL: https://doi.org/10.4230/LIPIcs.CP.2024.25.
  30. Seyone Chithrananda. Seyonec/ChemBERTa-zinc-base-v1 ⋅ Hugging Face. URL: https://huggingface.co/seyonec/ChemBERTa-zinc-base-v1.
  31. Paul Shaw. Using constraint programming and local search methods to solve vehicle routing problems. In Michael J. Maher and Jean-Francois Puget, editors, Principles and Practice of Constraint Programming - CP98, 4th International Conference, Pisa, Italy, October 26-30, 1998, Proceedings, volume 1520 of Lecture Notes in Computer Science, pages 417-431. Springer, 1998. URL: https://doi.org/10.1007/3-540-49481-2_30.
  32. Jialin Song, ravi lanka, Yisong Yue, and Bistra Dilkina. A General Large Neighborhood Search Framework for Solving Integer Linear Programs. In Advances in Neural Information Processing Systems, volume 33, pages 20012-20023. Curran Associates, Inc., 2020. Google Scholar
  33. Nicolas Sonnerat, Pengming Wang, Ira Ktena, Sergey Bartunov, and Vinod Nair. Learning a Large Neighborhood Search Algorithm for Mixed Integer Programs. URL: https://doi.org/10.48550/arXiv.2107.10201.
  34. Filipe Souza, Diarmuid Grimes, and Barry O'Sullivan. Variable-Relationship Guided LNS for the Car Sequencing Problem. Springer Cham, 2023. URL: https://doi.org/10.1007/978-3-031-26438-2_34.
  35. Charles Thomas and Pierre Schaus. Revisiting the Self-adaptive Large Neighborhood Search. In Willem-Jan Van Hoeve, editor, Integration of Constraint Programming, Artificial Intelligence, and Operations Research, volume 10848, pages 557-566. Springer International Publishing, Cham, 2018. URL: https://doi.org/10.1007/978-3-319-93031-2_40.
  36. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, 2024. URL: https://doi.org/10.48550/arXiv.2412.13663.
  37. David Weininger. SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 28(1):31-36, February 1988. URL: https://doi.org/10.1021/ci00057a005.
  38. Yudong Xu, Wenhao Li, Scott Sanner, and Elias Boutros Khalil. Self-supervised transformers as iterative solution improvers for constraint satisfaction. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 69432-69450. PMLR, 2025. Google Scholar
  39. Yudong W. Xu, Wenhao Li, Scott Sanner, and Elias B. Khalil. Large neighborhood search meets iterative neural constraint heuristics, 2026. URL: https://arxiv.org/abs/2603.20801.
  40. Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik R. Narasimhan. COLLIE: Systematic Construction of Constrained Text Generation Tasks. In The Twelfth International Conference on Learning Representations, 2023. Google Scholar
  41. Chaoning Zhang, Chenshuang Zhang, Junha Song, John Seon Keun Yi, and In So Kweon. A Survey on Masked Autoencoder for Visual Self-supervised Learning. In Thirty-Second International Joint Conference on Artificial Intelligence, volume 6, pages 6805-6813, 2023. URL: https://doi.org/10.24963/ijcai.2023/762.
  42. Honghua Zhang, Po-Nien Kung, Masahiro Yoshida, Guy Van den Broeck, and Nanyun Peng. Adaptable Logical Control for Large Language Models. Advances in Neural Information Processing Systems, 37:115563-115587, December 2024. Google Scholar
  43. Jianan Zhou, Yaoxin Wu, Zhiguang Cao, Wen Song, Jie Zhang, and Zhenghua Chen. Learning Large Neighborhood Search for Vehicle Routing in Airport Ground Handling. IEEE Transactions on Knowledge and Data Engineering, PP, 2023. URL: https://doi.org/10.1109/TKDE.2023.3249799.
  44. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A Benchmarking Platform for Text Generation Models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18, pages 1097-1100, New York, NY, USA, 2018. Association for Computing Machinery. URL: https://doi.org/10.1145/3209978.3210080.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact