Document Open Access Logo

Algorithms for Computing Very Large BWTs: a Short Survey

Authors Diego Díaz-Domínguez , Lavinia Egidi , Veronica Guerrini , Felipe A. Louza , Giovanna Rosone

PDF HTML 5

Files

Thumbnail PDF
PDF
OASIcs.Manzini.7.pdf
  • Filesize: 0.93 MB
  • 28 pages
HTML5 Logo HTML (experimental)
OASIcs.Manzini.7.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Data structures design and analysis
Keywords
  • Burrows-Wheeler transform
  • Extended Burrows-Wheeler transform
  • external memory
  • text compression
  • longest common prefix

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

The Burrows-Wheeler Transform (BWT) is a fundamental string transformation that, although initially introduced for data compression, has been extensively utilized across various domains, including text indexing and pattern matching within large datasets. Although the BWT construction is linear, the constants make the task impractical for large datasets, and as highlighted by Ferragina et al. [Paolo Ferragina et al., 2012], "to use it, one must first build it!". Thus, the construction of the BWT remains a significant challenge. For these reasons, during the past three decades there has been a succession of new algorithms for its construction using techniques that work in external memory or that use text compression. In this survey, we revise some of the most important advancements and tools presented in the past years for computing large BWTs exploiting external memory or text compression approaches without using additional information about the data.

Cite As Get BibTex

Diego Díaz-Domínguez, Lavinia Egidi, Veronica Guerrini, Felipe A. Louza, and Giovanna Rosone. Algorithms for Computing Very Large BWTs: a Short Survey. In The Expanding World of Compressed Data: A Festschrift for Giovanni Manzini's 60th Birthday. Open Access Series in Informatics (OASIcs), Volume 131, pp. 7:1-7:28, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.Manzini.7

Author Details

Diego Díaz-Domínguez
  • Department of Computer Science, University of Helsinki, Finland
Lavinia Egidi
  • Computer Science Institute, DiSIT, University of Eastern Piedmont, Vercelli, Italy
Veronica Guerrini
  • Department of Computer Science, University of Pisa, Italy
Felipe A. Louza
  • Faculty of Electrical Engineering, Federal University of Uberlândia, Brazil
Giovanna Rosone
  • Department of Computer Science, University of Pisa, Italy

Funding

  • Díaz-Domínguez, Diego: Supported by the European Union’s Horizon Europe research and innovation programme under grant agreement No 101060011.
  • Guerrini, Veronica: Supported by the Next Generation EU PNRR MUR M4 C2 Inv 1.5 project ECS00000017 - "THE - Tuscany Health Ecosystem" - Spoke 6 "Precision medicine & personalized healthcare" CUP I53C22000780001.
  • Louza, Felipe A.: Supported by Brazilian agencies CNPq (grants 408314/2023-0 and 311128/2025-4) and FAPEMIG (grant APQ-01217-22).
  • Rosone, Giovanna: Partially supported by the Next Generation EU PNRR MUR M4 C2 Inv 1.5 project ECS00000017 - "THE - Tuscany Health Ecosystem" - Spoke 6 "Precision medicine & personalized healthcare" CUP I53C22000780001, by the MUR PRIN 2022YRB97K PINC and by project "Hub multidisciplinare e interregionale di ricerca e sperimentazione clinica per il contrasto alle pandemie e all’antibioticoresistenza (PAN-HUB)" funded by the Italian Ministry of Health (POS 2014–2020, project ID: T4-AN-07, CUP: I53C22001300001).

References

  1. Donald Adjeroh, Timothy Bell, and Amar Mukherjee. The Burrows-Wheeler transform: data compression, suffix arrays, and pattern matching. Springer, Boston, MA, 2008. Google Scholar
  2. Hideo Bannai, Travis Gagie, et al. Refining the r-index. Theoretical Computer Science, 812:96-108, 2020. URL: https://doi.org/10.1016/J.TCS.2019.08.005.
  3. Hideo Bannai, Juha Kärkkäinen, Dominik Köppl, and Marcin Piątkowski. Constructing the Bijective and the Extended Burrows-Wheeler Transform in Linear Time. In Proc. 32nd Annual Symposium on Combinatorial Pattern Matching (CPM), pages 7:1-7:16, 2021. URL: https://doi.org/10.4230/LIPIcs.CPM.2021.7.
  4. Jérémy Barbay, Travis Gagie, Gonzalo Navarro, and Yakov Nekrich. Alphabet partitioning for compressed rank/select and applications. In Proc. 21st International Symposium on Algorithms and Computation (ISAAC), pages 315-326, 2010. URL: https://doi.org/10.1007/978-3-642-17514-5_27.
  5. Markus J. Bauer, Anthony J. Cox, and Giovanna Rosone. Lightweight algorithms for constructing and inverting the BWT of string collections. Theoretical Computer Science, 483(0):134-148, 2013. URL: https://doi.org/10.1016/j.tcs.2012.02.002.
  6. Gianmarco Bertola, Anthony J. Cox, Veronica Guerrini, and Giovanna Rosone. A Class of Heuristics for Reducing the Number of BWT-Runs in the String Ordering Problem. In Proc. 35th Annual Symposium on Combinatorial Pattern Matching (CPM), pages 7:1-7:15, 2024. URL: https://doi.org/10.4230/LIPIcs.CPM.2024.7.
  7. Nico Bertram, Johannes Fischer, and Lukas Nalbach. Move-r: optimizing the r-index. In Proc. 22nd International Symposium on Experimental Algorithms (SEA), pages 1:1-1:19, 2024. URL: https://doi.org/10.4230/LIPICS.SEA.2024.1.
  8. Paola Bonizzoni, Gianluca Della Vedova, Yuri Pirola, Marco Previtali, and Raffaella Rizzi. Computing the multi-string BWT and LCP array in external memory. Theoretical Computer Science, 862:42-58, 2021. URL: https://doi.org/10.1016/j.tcs.2020.11.041.
  9. Christina Boucher, Davide Cenzato, Zsuzsanna Lipták, Massimiliano Rossi, and Marinella Sciortino. Computing the original eBWT faster, simpler, and with less memory. In Proc. 28th International Symposium on String Processing and Information Retrieval (SPIRE), pages 129-142, 2021. URL: https://doi.org/10.1007/978-3-030-86692-1_11.
  10. Christina Boucher, Davide Cenzato, Zsuzsanna Lipták, Massimiliano Rossi, and Marinella Sciortino. Computing the original eBWT faster, simpler, and with less memory. In Proceedings of 28th International Symposium in String Processing and Information Retrieval SPIRE 2021, volume 12944 of Lecture Notes in Computer Science, pages 129-142, 2021. URL: https://doi.org/10.1007/978-3-030-86692-1_11.
  11. Christina Boucher, Travis Gagie, Alan Kuhnle, Ben Langmead, Giovanni Manzini, and Taher Mun. Prefix-free parsing for building big BWTs. Algorithms for Molecular Biology, 14(1):13:1-13:15, 2019. URL: https://doi.org/10.1186/S13015-019-0148-5.
  12. Michael Burrows and David J. Wheeler. A block sorting lossless data compression algorithm. Technical Report 124, Digital Equipment Corporation, 1994. Google Scholar
  13. Davide Cenzato, Veronica Guerrini, Zsuzsanna Lipták, and Giovanna Rosone. Computing the optimal BWT of very large string collections. In Proc. 33rd Data Compression Conference (DCC), pages 71-80, 2023. URL: https://doi.org/10.1109/DCC55655.2023.00015.
  14. Davide Cenzato and Zsuzsanna Lipták. A survey of BWT variants for string collections. Bioinformatics, 40(7):btae333, 2024. URL: https://doi.org/10.1093/bioinformatics/btae333.
  15. Davide Cenzato, Zsuzsanna Lipták, Nadia Pisanti, Giovanna Rosone, and Marinella Sciortino. BWT for string collections. submitted to Festschrift’s honoree Giovanni Manzini. Google Scholar
  16. Moses Charikar, Eric Lehman, Ding Liu, Rina Panigrahy, Manoj Prabhakaran, Amit Sahai, and Abhi Shelat. The smallest grammar problem. IEEE Transactions on Information Theory, 51(7):2554-2576, 2005. URL: https://doi.org/10.1109/TIT.2005.850116.
  17. Dustin Cobas, Travis Gagie, and Gonzalo Navarro. A fast and small subsampled r-index. In Proc. 32nd Annual Symposium on Combinatorial Pattern Matching (CPM), page article 13, 2021. Google Scholar
  18. The Computational Pan-Genomics Consortium. Computational pan-genomics: status, promises and challenges. Briefings in Bioinformatics, 19(1):118-135, 2016. URL: https://doi.org/10.1093/bib/bbw089.
  19. Anthony J. Cox, Markus J. Bauer, Tobias Jakobi, and Giovanna Rosone. Large-scale compression of genomic sequence databases with the Burrows–Wheeler transform. Bioinformatics, 28(11):1415-1419, 2012. URL: https://doi.org/10.1093/bioinformatics/bts173.
  20. Anthony J. Cox, Fabio Garofalo, Giovanna Rosone, and Marinella Sciortino. Lightweight LCP construction for very large collections of strings. Journal of Discrete Algorithms, 37:17-33, 2016. URL: https://doi.org/10.1016/J.JDA.2016.03.003.
  21. Diego Diaz-Dominguez. Efficient terabyte-scale text compression via stable local consistency and parallel grammar processing. arXiv preprint arXiv:2411.12439, 2024. to appear in Proc. 23rd Symposioum on Experimental Algorithms (SEA 2025). Google Scholar
  22. Diego Díaz-Domínguez and Gonzalo Navarro. Efficient construction of the BWT for repetitive text using string compression. Information and Computation, 294:105088, 2023. URL: https://doi.org/10.1016/j.ic.2023.105088.
  23. Lavinia Egidi, Felipe A. Louza, Giovanni Manzini, and Guilherme P. Telles. External memory BWT and LCP computation for sequence collections with applications. Algorithms for Molecular Biology, 14(1):6:1-6:15, 2019. URL: https://doi.org/10.1186/S13015-019-0140-0.
  24. Lavinia Egidi and Giovanni Manzini. Lightweight merging of compressed indices based on BWT variants. Theoretical Computer Science, 812:214-229, 2020. URL: https://doi.org/10.1016/j.tcs.2019.11.001.
  25. Paolo Ferragina. Pearls of Algorithm Engineering. Cambridge University Press, 2023. Google Scholar
  26. Paolo Ferragina, Travis Gagie, and Giovanni Manzini. Lightweight data indexing and compression in external memory. Algorithmica, 63(3):707-730, 2012. URL: https://doi.org/10.1007/S00453-011-9535-0.
  27. Paolo Ferragina and Giovanni Manzini. An experimental study of a compressed index. Information Sciences, 135(1):13-28, 2001. URL: https://doi.org/10.1016/S0020-0255(01)00098-6.
  28. Travis Gagie, Gonzalo Navarro, and Nicola Prezza. Fully functional suffix trees and optimal text searching in bwt-runs bounded space. Journal of the ACM (JACM), 67(1):1-54, 2020. URL: https://doi.org/10.1145/3375890.
  29. Alexander Golynski. Rank/select operations on large alphabets: a tool for text. In Proc. 17th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), volume 122, page 368, 2006. Google Scholar
  30. Roberto Grossi, Ankur Gupta, Jeffrey Scott Vitter, et al. High-order entropy-compressed text indexes. In Proc. 14th annual ACM-SIAM symposium on Discrete algorithms (SODA), volume 3, pages 841-850, 2003. Google Scholar
  31. Veronica Guerrini, Felipe A. Louza, and Giovanna Rosone. Parallel lossy compression for large FASTQ files. In 16th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC), pages 97-120, 2023. URL: https://doi.org/10.1007/978-3-031-38854-5_6.
  32. James Holt and Leonard McMillan. Merging of multi-string BWTs with applications. Bioinformatics, 30(24):3524-3531, 2014. URL: https://doi.org/10.1093/bioinformatics/btu584.
  33. Hongwei Huo, Pengfei Liu, Chenhui Wang, Hongbo Jiang, and Jeffrey Scott Vitter. CIndex: compressed indexes for fast retrieval of FASTQ files. Bioinformatics, 38(2):335-343, 2021. URL: https://doi.org/10.1093/bioinformatics/btab655.
  34. Bruce Jacob, David Wang, and Spencer Ng. Memory systems: cache, DRAM, disk. Morgan Kaufmann, 2010. Google Scholar
  35. Lilian Janin, Giovanna Rosone, and Anthony J. Cox. Adaptive reference-free compression of sequence quality scores. Bioinformatics, 30(1):24-30, 2014. URL: https://doi.org/10.1093/bioinformatics/btt257.
  36. Juha Kärkkäinen, Giovanni Manzini, and Simon J. Puglisi. Permuted longest-common-prefix array. In Proc. 20th Annual Symposium on Combinatorial Pattern Matching (CPM), pages 181-192, 2009. URL: https://doi.org/10.1007/978-3-642-02441-2_17.
  37. Juha Kärkkäinen, Peter Sanders, and Stefan Burkhardt. Linear work suffix array construction. Journal of the ACM (JACM), 53(6):918-936, 2006. URL: https://doi.org/10.1145/1217856.1217858.
  38. Richard M. Karp and Michael O. Rabin. Efficient randomized pattern-matching algorithms. IBM Journal of Research and Development, 31(2):249-260, 1987. URL: https://doi.org/10.1147/rd.312.0249.
  39. John C. Kieffer and En Hui Yang. Grammar-based codes: a new class of universal lossless source codes. IEEE Transactions on Information Theory, 46(3):737-754, 2000. URL: https://doi.org/10.1109/18.841160.
  40. Pang Ko and Srinivas Aluru. Space efficient linear time construction of suffix arrays. Journal of Discrete Algorithms, 3(2-4):143-156, 2005. URL: https://doi.org/10.1016/j.jda.2004.08.002.
  41. Tak Wah Lam, Wing-Kin Sung, Siu-Lung Tam, Chi-Kwong Wong, and Siu-Ming Yiu. Compressed indexing and local alignment of DNA. Bioinformatics, 24(6):791-797, 2008. URL: https://doi.org/10.1093/bioinformatics/btn032.
  42. Ben Langmead, Cole Trapnell, Mihai Pop, and Steven L. Salzberg. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome biology, 10:1-10, 2009. URL: https://doi.org/10.1186/gb-2009-10-3-r25.
  43. Heng Li. Exploring single-sample SNP and INDEL calling with whole-genome de novo assembly. Bioinformatics, 28(14):1838-1844, 2012. URL: https://doi.org/10.1093/bioinformatics/bts280.
  44. Heng Li. Fast construction of FM-index for long sequence reads. Bioinformatics, 30(22):3274-3275, 2014. URL: https://doi.org/10.1093/bioinformatics/btu541.
  45. Heng Li. BWT construction and search at the terabase scale. Bioinformatics, 40(12):btae717, 2024. URL: https://doi.org/10.1093/bioinformatics/btae717.
  46. Heng Li and Richard Durbin. Fast and accurate long-read alignment with Burrows-Wheeler transform. Bioinformatics, 26(5):589-595, 2010. URL: https://doi.org/10.1093/bioinformatics/btp698.
  47. Zsuzsanna Lipták, Francesco Masillo, and Simon J Puglisi. Suffix sorting via matching statistics. Algorithms for Molecular Biology, 19(1):11, 2024. URL: https://doi.org/10.1186/s13015-023-00245-z.
  48. Felipe A. Louza, Simon Gog, and Guilherme P. Telles. Inducing enhanced suffix arrays for string collections. Theoretical Computer Science, 678:22-39, 2017. URL: https://doi.org/10.1016/J.TCS.2017.03.039.
  49. Felipe A. Louza, Guilherme P. Telles, Steve Hoffmann, and Cristina Dutra de Aguiar Ciferri. Generalized enhanced suffix array construction in external memory. Algorithms for Molecular Biology, 12(1):26:1-26:16, 2017. URL: https://doi.org/10.1186/S13015-017-0117-9.
  50. Udi Manber and Eugene W. Myers. Suffix arrays: A new method for on-line string searches. SIAM Journal of Computation., 22(5):935-948, 1993. URL: https://doi.org/10.1137/0222058.
  51. Sabrina Mantaci, Antonio Restivo, Giovanna Rosone, and Marinella Sciortino. An extension of the Burrows-Wheeler transform. Theoretical Computer Science, 387(3):298-312, 2007. URL: https://doi.org/10.1016/j.tcs.2007.07.014.
  52. Francesco Masillo. Matching Statistics Speed up BWT Construction. In Proc. 31st Annual European Symposium on Algorithms (ESA 2023), pages 83:1-83:15, 2023. URL: https://doi.org/10.4230/LIPIcs.ESA.2023.83.
  53. Takaaki Nishimoto and Yasuo Tabei. Optimal-Time Queries on BWT-Runs Compressed Indexes. In Proc. 48th International Colloquium on Automata, Languages, and Programming (ICALP), volume 198, pages 101:1-101:15, 2021. URL: https://doi.org/10.4230/LIPICS.ICALP.2021.101.
  54. Ge Nong. Practical linear-time o(1)-workspace suffix sorting for constant alphabets. ACM Transactions on Information Systems, 31(3):15:1-15:15, 2013. URL: https://doi.org/10.1145/2493175.2493180.
  55. Ge Nong, Wai Hong Chan, Sheng Qing Hu, and Yi Wu. Induced sorting suffixes in external memory. ACM Transactions on Information Systems (TOIS), 33(3):1-15, 2015. URL: https://doi.org/10.1145/2699665.
  56. Ge Nong, Sen Zhang, and Wai Hong Chan. Two efficient algorithms for linear time suffix array construction. IEEE Transactions on Computers, 60(10):1471-1484, 2011. URL: https://doi.org/10.1109/TC.2010.188.
  57. Daisuke Okanohara and Kunihiko Sadakane. A linear-time Burrows-Wheeler transform using induced sorting. In Proc. 16th International Symposium on String Processing and Information Retrieval (SPIRE), pages 90-101, 2009. URL: https://doi.org/10.1007/978-3-642-03784-9_9.
  58. Marco Oliva, Travis Gagie, and Christina Boucher. Recursive prefix-free parsing for building big bwts. In 2023 Data Compression Conference (DCC), pages 62-70, 2023. URL: https://doi.org/10.1109/DCC55655.2023.00014.
  59. Marco Oliva, Massimiliano Rossi, Jouni Sirén, Giovanni Manzini, Tamer Kahveci, Travis Gagie, and Christina Boucher. Efficiently merging r-indexes. In Proc. 31st Data Compression Conference (DCC), pages 203-212, 2021. URL: https://doi.org/10.1109/DCC50243.2021.00028.
  60. Nicola Prezza, Nadia Pisanti, Marinella Sciortino, and Giovanna Rosone. Variable-order reference-free variant discovery with the Burrows-Wheeler Transform. BMC Bioinformatics, 21-S(8):260, 2020. URL: https://doi.org/10.1186/S12859-020-03586-3.
  61. Yoshihiro Shibuya and Matteo Comin. Better quality score compression through sequence-based quality smoothing. BMC Bioinformatics, 20-S(9):302:1-302:11, 2019. URL: https://doi.org/10.1186/s12859-019-2883-5.
  62. Jared T. Simpson and Richard Durbin. Efficient de novo assembly of large genomes using compressed data structures. Genome Research, 22(3):549-56, 2012. URL: https://doi.org/10.1101/gr.126953.111.
  63. Jouni Sirén. Burrows-Wheeler Transform for Terabases. In Proc. 16th Data Compression Conference (DCC), pages 211-220, 2016. URL: https://doi.org/10.1109/DCC.2016.17.
  64. Jouni Sirén, Erik Garrison, Adam M Novak, Benedict Paten, and Richard Durbin. Haplotype-aware graph indexes. Bioinformatics, 36(2):400-407, 2020. URL: https://doi.org/10.1093/BIOINFORMATICS/BTZ575.
  65. Clare Turnbull, Richard H. Scott, Ellen Thomas, Louise Jones, Nirupa Murugaesu, Freya Boardman Pretty, Dina Halai, Emma Baple, Clare Craig, Angela Hamblin, Shirley Henderson, Christine Patch, Amanda O'Neill, Andrew Devereau, Katherine Smith, Antonio Rueda Martin, Alona Sosinsky, Ellen M. McDonagh, Razvan Sultana, Michael Mueller, Damian Smedley, Adam Toms, Lisa Dinh, Tom Fowler, Mark Bale, Tim Hubbard, Augusto Rendon, Sue Hill, and Mark J. Caulfield. The 100 000 genomes project: bringing whole genome sequencing to the nhs. BMJ, 361, 2018. URL: https://doi.org/10.1136/bmj.k1687.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact