Document Open Access Logo

Optimizing the Performance of the FM-Index for Large-Scale Data

Authors Eddie Ferro , Christina Boucher

PDF

File

Thumbnail PDF
PDF
OASIcs.Manzini.6.pdf
  • Filesize: 0.76 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Data compression
Keywords
  • FM-Index Acceleration
  • Run-Length Encoding
  • Suffix Array Optimization
  • Burrows-Wheeler Transform
  • Efficient Backward Search

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

The FM-index is a fundamental data structure used in bioinformatics to efficiently search for strings and index genomes. However, the FM-index can pose computational challenges, particularly in the context of large-scale genomic datasets, due to the complexity of its underlying components and data encodings. In this paper, we present a comprehensive review of efficient variants of the FM-index and the encoding strategies used to improve performance. We examine hardware-accelerated techniques, such as memory-efficient data layouts and cache-aware structures, as well as software-level innovations, including algorithmic refinements and compact representations. The reviewed work demonstrates substantial gains in both speed and scalability, making methods that use the FM-index more practical for high-throughput genomic applications. By analyzing the trade-offs and design choices of these variants, we highlight how combining hardware-aware and software-centric strategies enables more efficient FM-index construction and usage across a range of bioinformatics tasks.

Cite As Get BibTex

Eddie Ferro and Christina Boucher. Optimizing the Performance of the FM-Index for Large-Scale Data. In The Expanding World of Compressed Data: A Festschrift for Giovanni Manzini's 60th Birthday. Open Access Series in Informatics (OASIcs), Volume 131, pp. 6:1-6:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.Manzini.6

Author Details

Eddie Ferro
  • Department of Computer and Information Science and Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, FL, USA
Christina Boucher
  • Department of Computer and Information Science and Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, FL, USA

Acknowledgements

I want to thank Travis Gagie for helpful suggestions.

References

  1. Alberto Apostolico. The myriad virtues of subword trees. In Combinatorial Algorithms on Words, pages 85-96. Springer, 1985. Google Scholar
  2. Hideo Bannai, Travis Gagie, and Tomohiro I. Refining the r-index. Theoretical Computer Science, 812:96-108, April 2020. URL: https://doi.org/10.1016/j.tcs.2019.08.005.
  3. Djamal Belazzougui, Fabio Cunial, Juha Kärkkäinen, and Veli Mäkinen. Linear-time string indexing and analysis in small space. ACM Transactions on Algorithms (TALG), 16(2):1-54, 2020. URL: https://doi.org/10.1145/3381417.
  4. Nico Bertram, Johannes Fischer, and Lukas Nalbach. Move-r: optimizing the r-index. In 22nd International Symposium on Experimental Algorithms (SEA 2024), pages 1-1. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2024. Google Scholar
  5. Christina Boucher, Travis Gagie, Alan Kuhnle, Ben Langmead, Giovanni Manzini, and Taher Mun. Prefix-free parsing for building big BWTs. Algorithms Molecular Biology, 14(1):13:1-13:15, 2019. URL: https://doi.org/10.1186/S13015-019-0148-5.
  6. Christina Boucher, Dominik Köppl, Herman Perera, and Massimiliano Rossi. Accessing the suffix array via ϕ-1-forest. In International Symposium on String Processing and Information Retrieval, pages 86-98. Springer, 2022. URL: https://doi.org/10.1007/978-3-031-20643-6_7.
  7. Nathaniel K Brown, Travis Gagie, and Massimiliano Rossi. Rlbwt tricks. LIPIcs: Leibniz international proceedings in informatics, 233:16, 2022. Google Scholar
  8. Michael Burrows and David J. Wheeler. A block-sorting lossless data compression algorithm. Technical Report 124, Digital Equipment Corporation, Systems Research Center, Palo Alto, CA, May 1994. Google Scholar
  9. Dustin Cobas, Travis Gagie, and Gonzalo Navarro. A fast and small subsampled r-index. arXiv preprint arXiv:2103.15329, 2021. URL: https://arxiv.org/abs/2103.15329.
  10. J. J. Deng, W. K. Hon, D. Köppl, and K. Sadakane. Fm-indexing grammars induced by suffix sorting for long patterns. In Proceedings of the Data Compression Conference (DCC), pages 63-72. IEEE, 2022. URL: https://doi.org/10.1109/DCC52660.2022.00014.
  11. Diego Díaz-Domínguez, Saska Dönges, Simon J Puglisi, and Leena Salmela. Simple runs-bounded fm-index designs are fast. In 21st International Symposium on Experimental Algorithms (SEA 2023), pages 7-1. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2023. Google Scholar
  12. Martin Farach-Colton, Paolo Ferragina, and Shanmugavelayutham Muthukrishnan. On the sorting-complexity of suffix tree construction. Journal of the ACM (JACM), 47(6):987-1011, 2000. URL: https://doi.org/10.1145/355541.355547.
  13. P Ferragina and G Manzini. Indexing Compressed Text. Journal of the ACM, 52:552-581, 2005. URL: https://doi.org/10.1145/1082036.1082039.
  14. Paolo Ferragina and Giovanni Manzini. Opportunistic data structures with applications. In the Proceedings of the Foundations of Computer Science (FOCS), pages 390-398, 2000. URL: https://doi.org/10.1109/SFCS.2000.892127.
  15. Travis Gagie, Gonzalo Navarro, and Nicola Prezza. Fully Functional Suffix Trees and Optimal Text Searching in BWT-Runs Bounded Space. Journal of the ACM, 67(1):1-54, 2020. URL: https://doi.org/10.1145/3375890.
  16. Roberto Grossi, Ankur Gupta, Jeffrey Scott Vitter, et al. High-order entropy-compressed text indexes. In SODA, volume 3, pages 841-850, 2003. Google Scholar
  17. Aaron Hong, Marco Oliva, Dominik Köppl, Hideo Bannai, Christina Boucher, and Travis Gagie. Pfp-fm: an accelerated fm-index. Algorithms for Molecular Biology, 19(1):15, 2024. URL: https://doi.org/10.1186/S13015-024-00260-8.
  18. Juha Kärkkäinen, Giovanni Manzini, and Simon J Puglisi. Permuted longest-common-prefix array. In Proceedings of the Symposium on Combinatorial Pattern Matching (CPM), pages 181-192, 2009. URL: https://doi.org/10.1007/978-3-642-02441-2_17.
  19. Juha Kärkkäinen, Peter Sanders, and Stefan Burkhardt. Linear work suffix array construction. Journal of the ACM (JACM), 53(6):918-936, 2006. URL: https://doi.org/10.1145/1217856.1217858.
  20. B Langmead and S L Salzberg. Fast gapped-read alignment with Bowtie 2. Nature Methods, 9(4):357-359, 2012. Google Scholar
  21. Ben Langmead, Cole Trapnell, Mihai Pop, and Steven L. Salzberg. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biology, 10(3):R25-R25, 2009. Publisher: BioMed Central. URL: https://doi.org/10.1186/gb-2009-10-3-r25.
  22. Veli Mäkinen and Gonzalo Navarro. Succinct suffix arrays based on run-length encoding. In Annual Symposium on Combinatorial Pattern Matching, pages 45-56. Springer, 2005. URL: https://doi.org/10.1007/11496656_5.
  23. Udi Manber and Gene Myers. Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing, 22(5):935-948, 1993. URL: https://doi.org/10.1137/0222058.
  24. Edward M McCreight. A space-economical suffix tree construction algorithm. Journal of the ACM, 23(2):262-272, 1976. URL: https://doi.org/10.1145/321941.321946.
  25. J Ian Munro, Gonzalo Navarro, and Yakov Nekrich. Space-efficient construction of compressed indexes in deterministic linear time. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 408-424. SIAM, 2017. URL: https://doi.org/10.1137/1.9781611974782.26.
  26. V Mäkinen and G Navarro. Run-length FM-index. the Proceedings of the DIMACS Workshop: The Burrows-Wheeler Transform: Ten Years Later, pages 17-19, 2004. Google Scholar
  27. Gonzalo Navarro. Indexing text using the Ziv-Lempel trie. Journal of Discrete Algorithms, 2(1):87-114, 2004. URL: https://doi.org/10.1016/S1570-8667(03)00066-2.
  28. Gonzalo Navarro. Compact data structures: A practical approach. Cambridge University Press, 2016. Google Scholar
  29. Takaaki Nishimoto, Shunsuke Kanda, and Yasuo Tabei. An optimal-time rlbwt construction in bwt-runs bounded space. arXiv preprint arXiv:2202.07885, 2022. URL: https://arxiv.org/abs/2202.07885.
  30. Takaaki Nishimoto and Yasuo Tabei. Optimal-time queries on bwt-runs compressed indexes. arXiv preprint arXiv:2006.05104, 2020. Google Scholar
  31. Ge Nong, Sen Zhang, and Wai Hong Chan. Two efficient algorithms for linear time suffix array construction. IEEE transactions on computers, 60(10):1471-1484, 2010. URL: https://doi.org/10.1109/TC.2010.188.
  32. Daniel Saad Nogueira Nunes, Felipe Louza, Simon Gog, Mauricio Ayala-Rincón, and Gonzalo Navarro. A grammar compression algorithm based on induced suffix sorting. In 2018 Data Compression Conference, pages 42-51. IEEE, 2018. URL: https://doi.org/10.1109/DCC.2018.00012.
  33. Pizza & Chili repetitive corpus. Available at http://pizzachili.dcc.uchile.cl/repcorpus.html. Accessed 16 April 2020.
  34. Alberto Policriti and Nicola Prezza. Lz77 computation based on the run-length encoded bwt. Algorithmica, 80(7):1986-2011, 2018. URL: https://doi.org/10.1007/S00453-017-0327-Z.
  35. Nicola Prezza. Compressed computation for text indexing, 2017. Google Scholar
  36. Kunihiko Sadakane. Compressed text databases with efficient query algorithms based on the compressed suffix array. In International symposium on algorithms and computation, pages 410-421. Springer, 2000. URL: https://doi.org/10.1007/3-540-40996-3_35.
  37. J Sirén. Compressed suffix arrays for massive data. In International Symposium on String Processing and Information Retrieval, pages 63-74, 2009. Google Scholar
  38. Esko Ukkonen. On-line construction of suffix trees. Algorithmica, 14(3):249-260, 1995. URL: https://doi.org/10.1007/BF01206331.
  39. Peter Weiner. Linear pattern matching algorithms. In Proceedings of the Annual Symposium on Switching and Automata Theory (SWAT), pages 1-11. IEEE, 1973. URL: https://doi.org/10.1109/SWAT.1973.13.
  40. Mohsen Zakeri, Nathaniel K Brown, Omar Y Ahmed, Travis Gagie, and Ben Langmead. Movi: a fast and cache-efficient full-text pangenome index. iScience, 27(12), 2024. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact