Document Open Access Logo

Efficient Terabyte-Scale Text Compression via Stable Local Consistency and Parallel Grammar Processing

Author Diego Díaz-Domínguez

PDF HTML 5

Files

Thumbnail PDF
PDF
LIPIcs.SEA.2025.14.pdf
  • Filesize: 1.13 MB
  • 20 pages
HTML5 Logo HTML (experimental)
LIPIcs.SEA.2025.14.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Data compression
Keywords
  • Grammar compression
  • locally consistent parsing
  • hashing

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

We present compression algorithms designed to process terabyte-sized datasets in parallel. Our approach builds on locally consistent grammars, a lightweight form of compression, combined with simple post-processing techniques to achieve further space reductions. Locally consistent grammar algorithms are suitable for scaling as they need minimal satellite information to compact the text, but they are not inherently parallel. To enable parallelisation, we introduce a novel concept that we call stable local consistency. A grammar algorithm ALG is stable if for any pattern P occurring in a collection 𝒯 = {T_1, T_2, …, T_k}, instances ALG(T_1), ALG(T_2), …, ALG(T_k) independently produce cores for P with the same topology. In a locally consistent grammar, the core of P is a subset of nodes and edges in the parse tree of 𝒯 that remains the same in all the occurrences of P. This feature enables compression, but it only holds if ALG defines a common set of nonterminal symbols for the strings. Stability removes this restriction, allowing us to run ALG(T_1), ALG(T_2), …, ALG(T_k) in parallel and subsequently merge their grammars into a single output equivalent to that of ALG(𝒯). We implemented our ideas and tested them on massive datasets. Our experiments showed that our method could process 7.9 TB of bacterial genomes in around nine hours, using 16 threads and 0.43 bits/symbol of working memory, achieving a compression ratio of 85x.

Cite As Get BibTex

Diego Díaz-Domínguez. Efficient Terabyte-Scale Text Compression via Stable Local Consistency and Parallel Grammar Processing. In 23rd International Symposium on Experimental Algorithms (SEA 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 338, pp. 14:1-14:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.SEA.2025.14

Author Details

Diego Díaz-Domínguez
  • University of Helsinki, Finland

Funding

This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101060011.

References

  1. Tuğkan Batu and S Cenk Sahinalp. Locally consistent parsing and applications to approximate string comparisons. In Proc. 9th International Conference on Developments in Language Theory (DLT), pages 22-35, 2005. Google Scholar
  2. Or Birenzwige, Shay Golan, and Ely Porat. Locally consistent parsing for text indexing in small space. In Proc. 31th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 607-626, 2020. URL: https://doi.org/10.1137/1.9781611975994.37.
  3. Christina Boucher, Travis Gagie, Alan Kuhnle, Ben Langmead, Giovanni Manzini, and Taher Mun. Prefix-free parsing for building big BWTs. Algorithms for Molecular Biology, 14:1-15, 2019. URL: https://doi.org/10.1186/S13015-019-0148-5.
  4. Moses Charikar, Eric Lehman, Ding Liu, Rina Panigrahy, Manoj Prabhakaran, Amit Sahai, and Abhi Shelat. The smallest grammar problem. IEEE Transactions on Information Theory, 51(7):2554-2576, 2005. URL: https://doi.org/10.1109/TIT.2005.850116.
  5. Anders Roy Christiansen, Mikko Berggren Ettienne, Tomasz Kociumaka, Gonzalo Navarro, and Nicola Prezza. Optimal-time dictionary-compressed indexes. ACM Transactions on Algorithms (TALG), 17(1):1-39, 2020. URL: https://doi.org/10.1145/3426473.
  6. Richard Cole and Uzi Vishkin. Deterministic coin tossing and accelerating cascades: micro and macro techniques for designing parallel algorithms. In Proc. 18th Annual ACM Symposium on Theory of Computing (STOC), pages 206-219, 1986. URL: https://doi.org/10.1145/12130.12151.
  7. Jin-Jie Deng, Wing-Kai Hon, Dominik Köppl, and Kunihiko Sadakane. FM-indexing grammars induced by suffix sorting for long patterns. In Proc. 22nd Data Compression Conference (DCC), pages 63-72. IEEE, 2022. URL: https://doi.org/10.1109/DCC52660.2022.00014.
  8. Sebastian Deorowicz, Agnieszka Danek, and Heng Li. AGC: compact representation of assembled genomes with fast queries and updates. Bioinformatics, 39(3):btad097, 2023. URL: https://doi.org/10.1093/BIOINFORMATICS/BTAD097.
  9. Diego Díaz-Domínguez and Gonzalo Navarro. Efficient construction of the BWT for repetitive text using string compression. Information and Computation, 294:105088, 2023. URL: https://doi.org/10.1016/J.IC.2023.105088.
  10. Diego Díaz-Domínguez, Gonzalo Navarro, and Alejandro Pacheco. An LMS-based grammar self-index with local consistency properties. In Proc. 28th International Symposium on String Processing and Information Retrieval (SPIRE), pages 100-113, 2021. URL: https://doi.org/10.1007/978-3-030-86692-1_9.
  11. Diego Díaz-Domínguez and Leena Salmela. Computing all-vs-all MEMs in grammar-compressed text. In Proc. 30th International Symposium on String Processing and Information Retrieval (SPIRE), pages 157-170. Springer, 2023. URL: https://doi.org/10.1007/978-3-031-43980-3_13.
  12. Martin Dietzfelbinger, Joseph Gil, Yossi Matias, and Nicholas Pippenger. Polynomial hash functions are reliable. In Proc. 19th International Colloquium on Automata, Languages and Programming (ICALP), pages 235-246, 1992. Google Scholar
  13. Jarek Duda. Asymmetric numeral systems: entropy coding combining speed of huffman coding with compression rate of arithmetic coding. arXiv preprint arXiv:1311.2540, 2013. Google Scholar
  14. Travis Gagie, Tomohiro I, Giovanni Manzini, Gonzalo Navarro, Hiroshi Sakamoto, and Yoshimasa Takabatake. Rpair: Rescaling RePair with rsync. In Proc. 26th International Symposium on String Processing and Information Retrieval (SPIRE), pages 35-44, 2019. URL: https://doi.org/10.1007/978-3-030-32686-9_3.
  15. Paweł Gawrychowski, Adam Karczmarz, Tomasz Kociumaka, Jakub Łącki, and Piotr Sankowski. Optimal dynamic strings. In Proc. 29th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 1509-1528, 2018. Google Scholar
  16. David A. Huffman. A method for the construction of minimum-redundancy codes. Proceedings of the IRE, 40(9):1098-1101, 1952. Google Scholar
  17. Martin Hunt, Leandro Lima, Wei Shen, John Lees, and Zamin Iqbal. AllTheBacteria-all bacterial genomes assembled, available and searchable, 2024. bioRxiv preprint. URL: https://doi.org/10.1101/2024.03.08.584059.
  18. Artur Jeż. A really simple approximation of smallest grammar. Theoretical Computer Science, 616:141-150, 2016. URL: https://doi.org/10.1016/J.TCS.2015.12.032.
  19. Dominik Kempa and Tomasz Kociumaka. String synchronizing sets: sublinear-time BWT construction and optimal LCE data structure. In Proc. 51st Annual ACM Symposium on Theory of Computing (STOC), pages 756-767, 2019. URL: https://doi.org/10.1145/3313276.3316368.
  20. Dominik Kempa and Tomasz Kociumaka. Resolution of the burrows-wheeler transform conjecture. In Proc. 61st Annual Symposium on Foundations of Computer Science (FOCS), pages 1002-1013, 2020. URL: https://doi.org/10.1109/FOCS46700.2020.00097.
  21. Dominik Kempa and Tomasz Kociumaka. Dynamic suffix array with polylogarithmic queries and updates. In Proc. 54th Annual ACM Symposium on Theory of Computing (STOC), pages 1657-1670, 2022. URL: https://doi.org/10.1145/3519935.3520061.
  22. Dominik Kempa and Tomasz Kociumaka. Collapsing the hierarchy of compressed data structures: Suffix arrays in optimal compressed space. In Proc. 64th Annual Symposium on Foundations of Computer Science (FOCS), pages 1877-1886, 2023. URL: https://doi.org/10.1109/FOCS57990.2023.00114.
  23. John C. Kieffer and En Hui Yang. Grammar-based codes: a new class of universal lossless source codes. IEEE Transactions on Information Theory, 46(3):737-754, 2000. URL: https://doi.org/10.1109/18.841160.
  24. Tomasz Kociumaka, Gonzalo Navarro, and Francisco Olivares. Near-optimal search time in δ-optimal space, and vice versa. Algorithmica, 86(4):1031-1056, 2024. URL: https://doi.org/10.1007/S00453-023-01186-0.
  25. Tomasz Kociumaka, Gonzalo Navarro, and Nicola Prezza. Toward a definitive compressibility measure for repetitive sequences. IEEE Transactions on Information Theory, 69(4):2074-2092, 2022. URL: https://doi.org/10.1109/TIT.2022.3224382.
  26. N. Jesper Larsson and Alistair Moffat. Off-line dictionary-based compression. Proceedings of the IEEE, 88(11):1722-1732, 2000. URL: https://doi.org/10.1109/5.892708.
  27. Abraham Lempel and Jacob Ziv. On the complexity of finite sequences. IEEE Transactions on Information Theory, 22(1):75-81, 1976. URL: https://doi.org/10.1109/TIT.1976.1055501.
  28. Heng Li and Richard Durbin. Fast and accurate long-read alignment with Burrows-Wheeler Transform. Bioinformatics, 26(5):589-595, 2010. URL: https://doi.org/10.1093/BIOINFORMATICS/BTP698.
  29. Kurt Mehlhorn, Rajamani Sundar, and Christian Uhrig. Maintaining dynamic sequences under equality tests in polylogarithmic time. Algorithmica, 17:183-198, 1997. URL: https://doi.org/10.1007/BF02522825.
  30. S. Muthukrishnan and Süleyman Cenk Sahinalp. Approximate nearest neighbors and sequence comparison with block operations. In Proc of 32nd Annual ACM Symposium on Theory of Computing (STOC), pages 416-424, 2000. URL: https://doi.org/10.1145/335305.335353.
  31. Gonzalo Navarro. Computing MEMs and relatives on repetitive text collections. ACM Transactions on Algorithms, 21(1):1-33, 2024. URL: https://doi.org/10.1145/3701561.
  32. Craig G. Nevill-Manning and Ian H. Witten. Compression and explanation using hierarchical grammars. The Computer Journal, 40(2_and_3):103-116, 1997. Google Scholar
  33. Ge Nong. Practical linear-time O(1)-workspace suffix sorting for constant alphabets. ACM Transactions on Information Systems, 31(3):1-15, 2013. Google Scholar
  34. Ge Nong, Sen Zhang, and Wai Hong Chan. Linear suffix array construction by almost pure induced-sorting. In Proc. 19th Data Compression Conference (DCC), pages 193-202, 2009. URL: https://doi.org/10.1109/DCC.2009.42.
  35. Daniel Saad Nogueira Nunes, Felipe A. Louza, Simon Gog, Mauricio Ayala-Rincón, and Gonzalo Navarro. A grammar compression algorithm based on induced suffix sorting. In Proc. 28th Data Compression Conference (DCC), pages 42-51, 2018. URL: https://doi.org/10.1109/DCC.2018.00012.
  36. Carlos Ochoa and Gonzalo Navarro. RePair and all irreducible grammars are upper bounded by high-order empirical entropy. IEEE Transactions on Information Theory, 65(5):3160-3164, 2018. URL: https://doi.org/10.1109/TIT.2018.2871452.
  37. Sofya Raskhodnikova, Dana Ron, Ronitt Rubinfeld, and Adam Smith. Sublinear algorithms for approximating string compressibility. Algorithmica, 65:685-709, 2013. URL: https://doi.org/10.1007/S00453-012-9618-6.
  38. Sühleyman Cenk Sahinalp and Uzi Vishkin. Symmetry breaking for suffix tree construction. In Proc. 26th Annual ACM Symposium on Theory of Computing (STOC), pages 300-309, 1994. URL: https://doi.org/10.1145/195058.195164.
  39. Jacob Ziv and Abraham Lempel. A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3):337-343, 1977. URL: https://doi.org/10.1109/TIT.1977.1055714.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact