Document Open Access Logo

FM-Adaptive: A Practical Data-Aware FM-Index

Authors Hongwei Huo , Zongtao He , Pengfei Liu , Jeffrey Scott Vitter

PDF

File

Thumbnail PDF
PDF
OASIcs.Manzini.5.pdf
  • Filesize: 1.61 MB
  • 23 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Information systems → Information retrieval
  • Theory of computation → Design and analysis of algorithms
  • Theory of computation → Data structures design and analysis
  • Theory of computation → Data compression
  • Theory of computation → Pattern matching
Keywords
  • Text indexing
  • Burrows-Wheeler transform
  • Compressed wavelet trees
  • Entropy-compressed
  • Compressed data structures

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

The FM-index provides an important solution for efficient retrieval and search in textual big data. Its variants have been widely used in many fields including information retrieval, genome analysis, and web searching. In this paper, we propose improvements via a new compressed representation of the wavelet tree of the Burrows-Wheeler transform of the input text, which incorporates the gap γ-encoding. Our theoretical analysis shows that the new index, called FM-Adaptive, achieves asymptotic space optimality within a factor of 2 in the leading term, but it has a better compression and faster retrieval in practice than the competitive optimal compression boosting used in previous FM-indexes. We present a practical improved locate algorithm that provides substantially faster locating time based upon memoization, which takes advantage of the overlapping subproblems property. We design the lookup table for accelerated decoding to support fast pattern matching in a text. Extensive experiments demonstrate that FM-Adaptive provides faster query performance, often by a considerable amount, and/or comparable or better compression than other state-of-the-art FM-index methods.

Cite As Get BibTex

Hongwei Huo, Zongtao He, Pengfei Liu, and Jeffrey Scott Vitter. FM-Adaptive: A Practical Data-Aware FM-Index. In The Expanding World of Compressed Data: A Festschrift for Giovanni Manzini's 60th Birthday. Open Access Series in Informatics (OASIcs), Volume 131, pp. 5:1-5:23, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.Manzini.5

Author Details

Hongwei Huo
  • Department of Computer Science, Xidian University, Xi'an, China
Zongtao He
  • Department of Computer Science, Xidian University, Xi'an, China
Pengfei Liu
  • Department of Computer Science, Xidian University, Xi'an, China
Jeffrey Scott Vitter
  • Department of Computer Science, Tulane University, New Orleans, LA, USA
  • the University of Mississippi, MS, USA

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62272358.

Acknowledgements

We would like to thank Simon Gog for sharing code.

Supplementary Materials

References

  1. Christina Boucher, Davide Cenzato, Zsuzsanna Lipták, Massimiliano Rossi, and Marinella Sciortino. r-indexing the eBWT. Information and Computation, 298:105155, 2024. URL: https://doi.org/10.1016/J.IC.2024.105155.
  2. Nathaniel K. Brown, Travis Gagie, Giovanni Manzini, Gonzalo Navarro, and Marinella Sciortino. Faster run-length compressed suffix arrays. In Alessio Conte, Andrea Marin, Giovanna Rosone, and Jeffrey S. Vitter, editors, From Strings to Graphs, and Back Again: A Festschrift for Roberto Grossi’s 60th Birthday. OASIcs, Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2025. Google Scholar
  3. M. Burrows and D. J. Wheeler. A block-sorting lossless data compression algorithm, 1994. Google Scholar
  4. Stefan Canzar and Steven L. Salzberg. Short read mapping: An algorithmic tour. Proceedings of the IEEE, 105(3):436-458, 2017. URL: https://doi.org/10.1109/JPROC.2015.2455551.
  5. Xiaoyang Chen, Hongwei Huo, Jun Huan, Jeffrey Scott Vitter, Weiguo Zheng, and Lei Zou. MSQ-Index: A succinct index for fast graph similarity search. IEEE Transactions on Knowledge and Data Engineering, 33(6):2654-2668, 2021. URL: https://doi.org/10.1109/TKDE.2019.2954527.
  6. David Richard Clark. Compact PAT trees. PhD thesis, University of Waterloo, Waterloo, Canada, 1996. Google Scholar
  7. Peter Elias. Efficient storage and retrieval by content and address of static files. J. Assoc. Comput. Mach., 21:246-260, 1974. URL: https://doi.org/10.1145/321812.321820.
  8. Peter Elias. Universal codeword sets and representations of the integers. IEEE Transactions on Information Theory, 21(2):194-203, 1975. URL: https://doi.org/10.1109/TIT.1975.1055349.
  9. Robert M. Fano. On the number of bits required to implement an associative memory. Computer Structures Group, MIT, Cambridge, MA, 1971. Google Scholar
  10. Paolo Ferragina and Giovanni Manzini. Opportunistic data structures with applications. In Proceedings of the 41st Annual IEEE Symposium on Foundations of Computer Science (FOCS'00), pages 390-398, 2000. URL: https://doi.org/10.1109/SFCS.2000.892127.
  11. Paolo Ferragina and Giovanni Manzini. Indexing compressed text. Journal of the ACM, 52(4):552-581, 2005. URL: https://doi.org/10.1145/1082036.1082039.
  12. Paolo Ferragina, Giovanni Manzini, Veli Mäkinen, and Gonzalo Navarro. An alphabet-friendly FM-index. In Proceedings of the 11th International Symposium on String Processing and Information Retrieval (SPIRE'04), pages 150-160, 2004. URL: https://doi.org/10.1007/978-3-540-30213-1_23.
  13. Paolo Ferragina, Giovanni Manzini, Veli Mäkinen, and Gonzalo Navarro. Compressed representations of sequences and full-text indexes. ACM Transactions on Algorithms, 3(2):Article 20, 2007. Google Scholar
  14. Paolo Ferragina, Giovanni Manzini, Veli Mäkinen, and Gonzalo Navarro. The myriad virtues of Wavelet Trees. Information and Computation, 207(8):849-866, 2009. URL: https://doi.org/10.1016/J.IC.2008.12.010.
  15. Luca Foschini, Roberto Grossi, Ankur Gupta, and Jeffrey Scott Vitter. When indexing equals compression: Experiments with compressing suffix arrays and applications. ACM Transactions on Algorithms, 2(4):611-639, 2006. Shorter versions appear in Proceedings of the 15th Annual SIAM/ACM Symposium on Discrete Algorithms (SODA '04), New Orleans, LA, January 2004, 636-645, and in "Fast compression with a static model in high-order entropy," Proceedings of IEEE Data Compression Conference, Snowbird, Utah, March 2004, 62-71. URL: https://doi.org/10.1145/1198513.1198521.
  16. Travis Gagie, Gonzalo Navarro, and Nicola Prezza. Fully functional suffix trees and optimal text searching in BWT-runs bounded space. Journal of the ACM, 67(1):Article 2, 2020. Google Scholar
  17. Raffaele Giancarlo, Giovanni Manzini, Antonio Restivo, Giovanna Rosone, and Marinella Sciortino. A new class of string transformations for compressed text indexing. Information and Computation, 294:105068, 2022. URL: https://doi.org/10.1016/J.IC.2023.105068.
  18. Simon Gog, Juha Kärkkäinen, Dominik Kempa, Matthias Petri, and Simon J. Puglisi. Fixed block compression boosting in FM-indexes: Theory and practice. Algorithmica, 81:1370-1391, 2019. URL: https://doi.org/10.1007/S00453-018-0475-9.
  19. Simon Gog, Gonzalo Navarro, and Matthias Petri. Improved and extended locating functionality on compressed suffix arrays. Journal of Discrete Algorithms, 32:53-63, 2015. URL: https://doi.org/10.1016/J.JDA.2015.01.006.
  20. Simon Gog and Matthias Petri. Optimized succinct data structures for massive data. Software: Practice and Experience, 44(11):1287-1314, 2014. URL: https://doi.org/10.1002/SPE.2198.
  21. Roberto Grossi, Ankur Gupta, and Jeffrey Scott Vitter. High-order entropy-compressed text indexes. In Proceedings of the 14th ACM-SIAM Symposium on Discrete Algorithms (SODA'03), pages 841-850, 2003. URL: http://dl.acm.org/citation.cfm?id=644108.644250.
  22. Roberto Grossi and Jeffrey Scott Vitter. Compressed suffix arrays and suffix trees with applications to text indexing and string matching. In Proceedings of the 32nd ACM Symposium on Theory of Computing (STOC'00), pages 397-406, 2000. Google Scholar
  23. Roberto Grossi and Jeffrey Scott Vitter. Compressed suffix arrays and suffix trees with applications to text indexing and string matching. SIAM Journal on Computing, 35(2):378-407, 2005. URL: https://doi.org/10.1137/S0097539702402354.
  24. Roberto Grossi, Jeffrey Scott Vitter, and Bojian Xu. Wavelet trees: From theory to practice. In Proceedings of the 1st International Conference on Data Compression, Communications and Processing, pages 210-221, 2011. URL: https://doi.org/10.1109/CCP.2011.16.
  25. Wing-Kai Hon, Rahul Shah, and Jeffrey Scott Vitter. Compression, indexing, and retrieval for massive string data. In Proceedings of the 21st annual conference on Combinatorial pattern matching (CPM'10), pages 260-274, 2010. URL: https://doi.org/10.1007/978-3-642-13509-5_24.
  26. David A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9):1098-1101, 1952. Google Scholar
  27. Hongwei Huo, Longgang Chen, Jeffrey Scott Vitter, and Yakov Nekrich. A practical implementation of compressed suffix arrays with applications to self-indexing. In Proceedings of the Data Compression Conference (DCC'14), pages 292-301, 2014. URL: https://doi.org/10.1109/DCC.2014.49.
  28. Hongwei Huo, Longgang Chen, Heng Zhao, Jeffrey Scott Vitter, Yakov Nekrich, and Qiang Yu. A Data-aware FM-index. In Proceedings of the 17th Workshop on Algorithm Engineering and Experiments (ALENEX'15), pages 10-23, 2015. URL: https://doi.org/10.1137/1.9781611973754.2.
  29. Hongwei Huo, Xiaoyang Chen, Xu Guo, and Jeffrey Scott Vitter. Efficient compression and indexing for highly repetitive DNA sequence collections. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 18(6):2394-2408, 2021. URL: https://doi.org/10.1109/TCBB.2020.2968323.
  30. Hongwei Huo, Zongtao He, Pengfei Liu, and Jeffrey Scott Vitter. FM-Adaptive: A data-aware FM-index [source code], January 2022. URL: https://doi.org/10.24433/CO.7967727.v1.
  31. Hongwei Huo, Pengfei Liu, Chenhui Wang, Hongbo Jiang, and Jeffrey Scott Vitter. CIndex: Compressed indexes for fast retrieval of FASTQ files. Bioinformatics, 38(2):335-343, 2022. URL: https://doi.org/10.1093/BIOINFORMATICS/BTAB655.
  32. Hongwei Huo, Peng Long, and Jeffrey Scott Vitter. Practical high-order entropy-compressed text self-indexing. IEEE Transactions on Knowledge and Data Engineering, 35(3):2943-2960, 2023. URL: https://doi.org/10.1109/TKDE.2021.3114401.
  33. Hongwei Huo, Zhigang Sun, Shuangjiang Li, Jeffrey Scott Vitter, and et al.. CS2A: A compressed suffix array-based method for short read alignment. In Proceedings of the Data Compression Conference (DCC'16), pages 271-278, 2016. Google Scholar
  34. Hongwei Huo, Yongze Yu, Zongtao He, and Jeffrey Scott Vitter. Indexing labeled property multidigraphs in entropy space, with applications. In Proceedings of the 41st IEEE International Conference on Data Engineering (ICDE'25), pages 2478-2492, 2025. Google Scholar
  35. Guy Jacobson. Space-efficient static trees and graphs. In Proceedings of the 30th Annual IEEE Symposium on Foundations of Computer Science (FOCS'89), pages 549-554, 1989. URL: https://doi.org/10.1109/SFCS.1989.63533.
  36. Juha Kärkkäinen and Simon J. Puglisi. Fixed block compression boosting in FM-indexes. In Proceedings of the 18th International Symposium on String Processing and Information Retrieval (SPIRE'11), pages 174-184, 2011. URL: https://doi.org/10.1007/978-3-642-24583-1_18.
  37. Juha Kärkkäinen, Dominik Kempa, and Simon J. Puglisi. Hybrid compression of bitvectors for the FM-index. In DCC, pages 302-311, 2014. URL: https://doi.org/10.1109/DCC.2014.87.
  38. Zhize Li, Jian Li, and Hongwei Huo. Optimal in-place suffix sorting. Information and Computation, 285(Part B):104818, 2022. URL: https://doi.org/10.1016/J.IC.2021.104818.
  39. Veli Mäkinen and Gonzalo Navarro. Succinct suffix arrays based on run-length encoding. In Proceedings of the 16th Annual Symposium on Combinatorial Pattern Matching (CPM), pages 45-56, 2005. LNCS 3537. URL: https://doi.org/10.1007/11496656_5.
  40. Veli Mäkinen and Gonzalo Navarro. Implicit compression boosting with applications to self-indexing. In Proceedings of the 14th International Symposium on String Processing and Information Retrieval (SPIRE'07), pages 229-241, 2007. URL: https://doi.org/10.1007/978-3-540-75530-2_21.
  41. Veli Mäkinen and Gonzalo Navarro. Dynamic entropy-compressed sequences and full-text indexes. ACM Transactions on Algorithms, 4(3):article 32, 2008. Google Scholar
  42. Veli Mäkinen, Gonzalo Navarro, Jouni Sirén, and Niko Välimäki. Storage and retrieval of highly repetitive sequence collections. J Comput Biol., 17(3):281-308, 2010. URL: https://doi.org/10.1089/CMB.2009.0169.
  43. Udi Manber and Gene Myers. Suffix arrays: A new method for on-line string searches. SIAM Journal on Computing, 22(5):935-948, 1993. URL: https://doi.org/10.1137/0222058.
  44. Giovanni Manzini. An analysis of the burrows-wheeler transform. Journal of the ACM, 48(3):407-430, 2001. URL: https://doi.org/10.1145/382780.382782.
  45. Edward M. McCreight. A space-economical suffix tree construction algorithm. Journal of the ACM, 23(2):262-272, 1976. URL: https://doi.org/10.1145/321941.321946.
  46. J. Ian Munro, Gonzalo Navarro, and Yakov Nekrich. Fast compressed self-indexes with deterministic linear-time construction. Algorithmica, 82(2):316-337, 2020. A conference version of this paper appeared in Proc. ISAAC 2017. URL: https://doi.org/10.1007/S00453-019-00637-X.
  47. Gonzalo Navarro. Indexing highly repetitive string collections, part ii: Compressed indexes. ACM Computing Surveys, 54(2):Article 26, 2021. Google Scholar
  48. Gonzalo Navarro. Compact Data Structures: A Practical Approach. Cambridge University Press, New York, USA, September 2016. Google Scholar
  49. Gonzalo Navarro and Veli Mäkinen. Compressed full-text indexes. ACM Computing Surveys, 39(1):Article 2, 2007. Google Scholar
  50. Gonzalo Navarro and Nicola Prezza. Universal compressed text indexing. Theoretical Computer Science, 762:41-50, 2019. URL: https://doi.org/10.1016/J.TCS.2018.09.007.
  51. Rajeev Raman, Venkatesh Raman, and S.Srinivasa Rao. Succinct indexable dictionaries with applications to encoding k-ary trees, prefix sums and multisets. ACM Transactions on Algorithms, 3(4):Article 43, 2007. URL: https://doi.org/10.1145/1290672.1290680.
  52. Kunihiko Sadakane. Compressed text databases with efficient query algorithms based on the compressed suffix array. In Proceedings of the 11th Symposium on Algorithms and Computation (ISAAC'00), pages 410-421, 2000. URL: https://doi.org/10.1007/3-540-40996-3_35.
  53. Kunihiko Sadakane. New text indexing functionalities of the compressed suffix arrays. Journal of Algorithms, 48(2):294-313, 2003. URL: https://doi.org/10.1016/S0196-6774(03)00087-7.
  54. Esko Ukkonen. On-line construction of suffix trees. Algorithmica, 14(3):249-260, 1995. URL: https://doi.org/10.1007/BF01206331.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact