Document Open Access Logo

Fast Kd-Trees for the Kullback-Leibler Divergence and Other Decomposable Bregman Divergences

Authors Tuyen Pham , Hubert Wagner

PDF

File

Thumbnail PDF
PDF
LIPIcs.WADS.2025.45.pdf
  • Filesize: 1.79 MB
  • 19 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Computational geometry
  • Information systems → Data structures
  • Mathematics of computing → Combinatorial algorithms
Keywords
  • Kd-tree
  • k-d tree
  • nearest neighbour search
  • Bregman divergence
  • decomposable Bregman divergence
  • KL divergence
  • relative entropy
  • cross entropy
  • Shannon’s entropy

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

The contributions of the paper span theoretical and implementational results. First, we prove that Kd-trees can be extended to ℝ^d with the distance measured by an arbitrary Bregman divergence. Perhaps surprisingly, this shows that the triangle inequality is not necessary for correct pruning in Kd-trees. Second, we offer an efficient algorithm and C++ implementation for nearest neighbour search for decomposable Bregman divergences.
The implementation supports the Kullback-Leibler divergence (relative entropy) which is a popular distance between probability vectors and is commonly used in statistics and machine learning. This is a step toward broadening the usage of computational geometry algorithms.
Our benchmarks show that our implementation efficiently handles both exact and approximate nearest neighbour queries. Compared to a linear search, we achieve two orders of magnitude speedup for practical scenarios in dimension up to 100. Our solution is simpler and more efficient than competing methods.

Cite As Get BibTex

Tuyen Pham and Hubert Wagner. Fast Kd-Trees for the Kullback-Leibler Divergence and Other Decomposable Bregman Divergences. In 19th International Symposium on Algorithms and Data Structures (WADS 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 349, pp. 45:1-45:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.WADS.2025.45

Author Details

Tuyen Pham
  • University of Florida, Gainesville, FL, USA
Hubert Wagner
  • University of Florida, Gainesville, Fl, USA

Funding

2022 Google Research Scholar Award in Algorithms and Optimization.

References

  1. Ahmed Abdelkader, Sunil Arya, Guilherme D da Fonseca, and David M Mount. Approximate nearest neighbor searching with non-euclidean and weighted distances. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 355-372. SIAM, 2019. Google Scholar
  2. Amirali Abdullah, John Moeller, and Suresh Venkatasubramanian. Approximate bregman near neighbors in sublinear time: beyond the triangle inequality. In Proceedings of the Twenty-Eighth Annual Symposium on Computational Geometry, SoCG '12, pages 31-40, New York, NY, USA, 2012. Association for Computing Machinery. URL: https://doi.org/10.1145/2261250.2261255.
  3. Marcel R. Ackermann and Johannes Blömer. Coresets and approximate clustering for bregman divergences. In Proceedings of the 2009 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1088-1097, 2009. URL: https://doi.org/10.1137/1.9781611973068.118.
  4. Martin Adamčík. The information geometry of bregman divergences and some applications in multi-expert reasoning. Entropy, 16(12):6338-6381, 2014. URL: https://doi.org/10.3390/e16126338.
  5. Sunil Arya and David M. Mount. Algorithms for fast vector quantization. In [Proceedings] DCC `93: Data Compression Conference, pages 381-390, 1993. URL: https://doi.org/10.1109/DCC.1993.253111.
  6. Sunil Arya, David M. Mount, and Onuttom Narayan. Accounting for Boundary Effects in Nearest Neighbor Searching. In Proceedings of the Eleventh Annual Symposium on Computational Geometry, SCG '95, pages 336-344, New York, NY, USA, 1995. Association for Computing Machinery. URL: https://doi.org/10.1145/220279.220315.
  7. Sunil Arya, David M. Mount, Nathan S. Netanyahu, Ruth Silverman, and Angela Y. Wu. An optimal algorithm for approximate nearest neighbor searching fixed dimensions. J. ACM, 45(6):891-923, November 1998. URL: https://doi.org/10.1145/293347.293348.
  8. M. Aumüller, E. Bernhardsson, and A. Faithfull. ANN-Benchmarks: A benchmarking tool for approximate nearest neighbor algorithms. Information Systems, 87:101-374, 2020. Google Scholar
  9. Arindam Banerjee, Srujana Merugu, Inderjit S. Dhillon, and Joydeep Ghosh. Clustering with Bregman divergences. Journal of Machine Learning Research, 6(58):1705-1749, 2005. URL: https://jmlr.org/papers/v6/banerjee05b.html.
  10. Heinz H Bauschke, Jonathan M Borwein, et al. Legendre functions and the method of random bregman projections. Journal of convex analysis, 4(1):27-67, 1997. Google Scholar
  11. Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Commun. ACM, 18:509-517, 1975. URL: https://doi.org/10.1145/361002.361007.
  12. Jon Louis Bentley. K-d trees for semidynamic point sets. In Proceedings of the Sixth Annual Symposium on Computational Geometry, SCG '90, pages 187-197, New York, NY, USA, 1990. Association for Computing Machinery. URL: https://doi.org/10.1145/98524.98564.
  13. Jean-Daniel Boissonnat, Frank Nielsen, and Richard Nock. Bregman Voronoi diagrams. Discrete and Computational Geometry, 44:281-307, 2010. URL: https://doi.org/10.1007/S00454-010-9256-1.
  14. Leonid Boytsov and Bilegsaikhan Naidan. Engineering efficient and effective non-metric space library. In Nieves R. Brisaboa, Oscar Pedreira, and Pavel Zezula, editors, Similarity Search and Applications - 6th International Conference, volume 8199 of Lecture Notes in Computer Science, pages 280-293. Springer, 2013. URL: https://doi.org/10.1007/978-3-642-41062-8_28.
  15. Leonid Boytsov and Bilegsaikhan Naidan. Learning to prune in metric and non-metric spaces. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. Google Scholar
  16. Lev M. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics, 7(3):200-217, 1967. Google Scholar
  17. L. Cayton. Bregman Proximity Search. Doctoral dissertation, UC San Diego, 2009. Google Scholar
  18. Lawrence Cayton. Bbtrees. URL: https://github.com/lcayton/bbtree.
  19. Lawrence Cayton. Fast nearest neighbor retrieval for bregman divergences. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pages 112-119, New York, NY, USA, 2008. Association for Computing Machinery. URL: https://doi.org/10.1145/1390156.1390171.
  20. Minh N Do and Martin Vetterli. Wavelet-based texture retrieval using generalized Gaussian density and Kullback-Leibler distance. IEEE Transactions on Image Processing, 11(2):146-158, 2002. URL: https://doi.org/10.1109/83.982822.
  21. H. Edelsbrunner and H. Wagner. Topological data analysis with bregman divergences. In "Proc. 33rd Ann. Symp. Comput. Geom., 2017", 1-16, pages 1-16, 2017. Google Scholar
  22. Jerome H. Friedman, Jon Louis Bentley, and Raphael Ari Finkel. An algorithm for finding best matches in logarithmic expected time. ACM Trans. Math. Softw., 3(3):209-226, September 1977. URL: https://doi.org/10.1145/355744.355745.
  23. Yikun Han, Chunjiang Liu, and Pengfei Wang. A comprehensive survey on vector database: Storage and retrieval technique, challenge. arXiv preprint arXiv:2310.11703, 2023. URL: https://doi.org/10.48550/arXiv.2310.11703.
  24. Piotr Indyk and Haike Xu. Worst-case performance of popular approximate nearest neighbor search implementations: guarantees and limitations. In Proceedings of the 37th International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2024. Curran Associates Inc. Google Scholar
  25. Fumitada Itakura. Analysis synthesis telephony based on the maximum likelihood method. Reports of the 6^th Int. Cong. Acoust., 1968. Google Scholar
  26. Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL: https://arxiv.org/abs/1312.6114.
  27. A. C. Koivunen and A. B. Kostinski. The feasibility of data whitening to improve performance of weather radar. Journal of Applied Meteorology, 38(6):741-749, 1999. URL: https://doi.org/10.1175/1520-0450(1999)038<0741:TFODWT>2.0.CO;2.
  28. Solomon Kullback and Richard A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951. Google Scholar
  29. Prasanta Chandra Mahalanobis. On the generalised distance in statistics. Proceedings of the National Institute of Sciences of India, 2(1):49-55, 1936. Retrieved 2016-09-27. Google Scholar
  30. Yu A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Trans. Pattern Anal. Mach. Intell., 42(4):824-836, April 2020. URL: https://doi.org/10.1109/TPAMI.2018.2889473.
  31. Yu A Malkov and Dmitry A Yashunin. Hnswlib - fast approximate nearest neighbor search. https://github.com/nmslib/hnswlib, 2023.
  32. Yury Malkov, Alexander Ponomarenko, Andrey Logvinov, and Vladimir Krylov. Approximate nearest neighbor algorithm based on navigable small world graphs. Information Systems, 45:61-68, 2014. URL: https://doi.org/10.1016/J.IS.2013.10.006.
  33. MaridDB. Mariadb vector. URL: https://mariadb.org/projects/mariadb-vector/.
  34. Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018. URL: https://doi.org/10.21105/JOSS.00861.
  35. MongoDB. Atlas Vector Search. URL: https://www.mongodb.com/products/platform/atlas-vector-search.
  36. David Mount. The ANN Programming Manual. Google Scholar
  37. Kevin P. Murphy. Machine learning: A Probabilistic Perspective. MIT press, 2012. Google Scholar
  38. Frank Nielsen and Richard Nock. The dual voronoi diagrams with respect to representational bregman divergences. In 2009 Sixth International Symposium on Voronoi Diagrams, pages 71-78, 2009. URL: https://doi.org/10.1109/ISVD.2009.15.
  39. Frank Nielsen and Richard Nock. The Bregman chord divergence. In 4th International Conference on Geometric Science of Information (GSI), pages 299-308. Springer, 2019. URL: https://doi.org/10.1007/978-3-030-26980-7_31.
  40. Frank Nielsen, Paolo Piro, and Michel Barlaud. Vptrees and bbtreepp. URL: https://github.com/FrankNielsen/FrankNielsen.github.io/tree/master/BregmanProximity.
  41. Frank Nielsen, Paolo Piro, and Michel Barlaud. Bregman vantage point trees for efficient nearest neighbor queries. In Proceedings - 2009 IEEE International Conference on Multimedia and Expo, (ICME), pages 878-881, 2009. URL: https://doi.org/10.1109/ICME.2009.5202635.
  42. Frank Nielsen, Paolo Piro, and Michel Barlaud. Tailored Bregman ball trees for effective nearest neighbors. In Proceedings of the 25th European Workshop on Computational Geometry (EuroCG), pages 29-32, 2009. Google Scholar
  43. R. Nock and F. Nielsen. Fitting the Smallest Enclosing Bregman Ball. In Machine Learning: ECML 2005, pages 649-656, October 2005. Google Scholar
  44. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011. URL: https://doi.org/10.5555/1953048.2078195.
  45. Alexander Ponomarenko, Nikita Avrelin, Bilegsaikhan Naidan, and Leonid Boytsov. Comparative analysis of data structures for approximate nearest neighbor search. In Proceedings of the Third International Conference on Data Analytics, January 2014. Google Scholar
  46. Ralph T. Rockafellar. Convex Analysis. Princeton University Press, 1970. Google Scholar
  47. Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948. URL: https://doi.org/10.1002/J.1538-7305.1948.TB01338.X.
  48. Yang Song, Yu Gu, Rui Zhang, and Ge Yu. BrePartition: Optimized High-Dimensional kNN Search with Bregman Distances, 2020. URL: https://arxiv.org/abs/2006.00227.
  49. Robert F. Sproull. Refinements to nearest-neighbor searching in k-dimensional trees. Algorithmica, 6:579-589, 1991. URL: https://doi.org/10.1007/BF01759061.
  50. Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11), 2008. Google Scholar
  51. Ruben Winastwan. Vector Indexing in Milvus. URL: https://zilliz.com/learn/how-to-pick-a-vector-index-in-milvus-visual-guide.
  52. Jun Zhang. Divergence Function, Duality, and Convex Analysis. Neural Computation, 16(1):159-195, January 2004. URL: https://doi.org/10.1162/08997660460734047.
  53. Zhenjie Zhang, Beng Chin Ooi, Srinivasan Parthasarathy, and Anthony K. H. Tung. Similarity Search on Bregman Divergence: Towards Non-Metric Indexing. Proc. VLDB Endow., 2(1):13-24, 2009. URL: https://doi.org/10.14778/1687627.1687630.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact