Document Open Access Logo

Bit Packed Encodings for Grammar-Compressed Strings Supporting Fast Random Access

Authors Alan M. Cleary , Joseph Winjum , Jordan Dood , Hiroki Shibata , Shunsuke Inenaga

PDF HTML 5

Files

Thumbnail PDF
PDF
LIPIcs.SEA.2025.12.pdf
  • Filesize: 1.22 MB
  • 17 pages
HTML5 Logo HTML (experimental)
LIPIcs.SEA.2025.12.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Theory of computation → Data compression
Keywords
  • String algorithms
  • data compression
  • random access
  • grammar-based compression

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Grammar-based compression is a powerful compression technique that allows for computation over the compressed data. While there has been extensive theoretical work on grammar and encoding size, there has been little work on practical grammar encodings. In this work, we consider the canonical array-of-arrays grammar representation and present a general bit packing approach for reducing its space requirements in practice. We then present three bit packing strategies based on this approach - one online and two offline - with different space-time trade-offs. This technique can be used to encode any grammar-compressed string while preserving the virtues of the array-of-arrays representation. We show that our encodings are Nlog₂ N away from the information-theoretic bound, where N is the number of symbols in the grammar, and that they are much smaller than methods that meet the information-theoretic bound in practice. Moreover, our experiments show that by using bit packed encodings we can achieve state-of-the-art performance both in grammar encoding size and run-time performance of random-access queries.

Cite As Get BibTex

Alan M. Cleary, Joseph Winjum, Jordan Dood, Hiroki Shibata, and Shunsuke Inenaga. Bit Packed Encodings for Grammar-Compressed Strings Supporting Fast Random Access. In 23rd International Symposium on Experimental Algorithms (SEA 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 338, pp. 12:1-12:17, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.SEA.2025.12

Author Details

Alan M. Cleary
  • National Center for Genome Resources, Santa Fe, NM, USA
Joseph Winjum
  • Montana State University, Bozeman, MT, USA
Jordan Dood
  • Hyalite Technologies LLC, Bozeman, MT, USA
Hiroki Shibata
  • Joint Graduate School of Mathematics for Innovation, Kyushu University, Fukuoka, Japan
Shunsuke Inenaga
  • Department of Informatics, Kyushu University, Fukuoka, Japan

Funding

The work of Alan M. Cleary, Joseph Winjum, and Jordan Dood was supported by NSF award number 2105391.
  • Inenaga, Shunsuke: The work of Shunsuke Inenaga was supported by JSPS KAKENHI grant numbers JP23K24808 and JP23K18466.

Acknowledgements

We thank the authors of [Travis Gagie et al., 2020] for providing the c1000 data set.

Supplementary Materials

References

  1. A061168 (partial sums of ⌊ log₂(k) ⌋). The On-Line Encyclopedia of Integer Sequences. URL: https://oeis.org/A061168.
  2. David Blackman and Sebastiano Vigna. Scrambled linear pseudorandom number generators. ACM Trans. Math. Softw., 47(4), September 2021. URL: https://doi.org/10.1145/3460772.
  3. Alan M. Cleary, Joseph Winjum, Jordan Dood, and Shunsuke Inenaga. Revisiting the folklore algorithm for random access to grammar-compressed strings. In Zsuzsanna Lipták, Edleno Moura, Karina Figueroa, and Ricardo Baeza-Yates, editors, String Processing and Information Retrieval, pages 88-101, Cham, 2024. Springer Nature Switzerland. URL: https://doi.org/10.1007/978-3-031-72200-4_7.
  4. P. Elias. Universal codeword sets and representations of the integers. IEEE Transactions on Information Theory, 21(2):194-203, 1975. URL: https://doi.org/10.1109/TIT.1975.1055349.
  5. Michael L. Fredman and Dan E. Willard. Surpassing the information theoretic bound with fusion trees. J. Comput. Syst. Sci., 47(3):424-436, 1993. URL: https://doi.org/10.1016/0022-0000(93)90040-4.
  6. Isamu Furuya, Takuya Takagi, Yuto Nakashima, Shunsuke Inenaga, Hideo Bannai, and Takuya Kida. Practical grammar compression based on maximal repeats. Algorithms, 13(4):103, 2020. URL: https://doi.org/10.3390/A13040103.
  7. Travis Gagie, Tomohiro I, Giovanni Manzini, Gonzalo Navarro, Hiroshi Sakamoto, Louisa Seelbach Benkner, and Yoshimasa Takabatake. Practical random access to SLP-compressed texts. In SPIRE 2020, volume 12303 of Lecture Notes in Computer Science, pages 221-231, 2020. URL: https://doi.org/10.1007/978-3-030-59212-7_16.
  8. Travis Gagie, Tomohiro I, Giovanni Manzini, Gonzalo Navarro, Hiroshi Sakamoto, and Yoshimasa Takabatake. Rpair: Rescaling RePair with rsync. In String Processing and Information Retrieval (SPIRE 2019), pages 35-44. Springer, 2019. URL: https://doi.org/10.1007/978-3-030-32686-9_3.
  9. Michał Gańczorz. Entropy Lower Bounds for Dictionary Compression. In Nadia Pisanti and Solon P. Pissis, editors, 30th Annual Symposium on Combinatorial Pattern Matching (CPM 2019), volume 128 of Leibniz International Proceedings in Informatics (LIPIcs), pages 11:1-11:18, Dagstuhl, Germany, 2019. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. URL: https://doi.org/10.4230/LIPIcs.CPM.2019.11.
  10. Simon Gog, Timo Beller, Alistair Moffat, and Matthias Petri. From theory to practice: Plug and play with succinct data structures. In Joachim Gudmundsson and Jyrki Katajainen, editors, Experimental Algorithms, pages 326-337, Cham, 2014. Springer International Publishing. URL: https://doi.org/10.1007/978-3-319-07959-2_28.
  11. N.J. Larsson and A. Moffat. Off-line dictionary-based compression. Proceedings of the IEEE, 88(11):1722-1732, 2000. URL: https://doi.org/10.1109/5.892708.
  12. D. Lemire and L. Boytsov. Decoding billions of integers per second through vectorization. Software: Practice and Experience, 45(1):1-29, 2015. URL: https://doi.org/10.1002/spe.2203.
  13. Shirou Maruyama, Yasuo Tabei, Hiroshi Sakamoto, and Kunihiko Sadakane. Fully-online grammar compression. In Oren Kurland, Moshe Lewenstein, and Ely Porat, editors, String Processing and Information Retrieval, pages 218-229, Cham, 2013. Springer International Publishing. URL: https://doi.org/10.1007/978-3-319-02432-5_25.
  14. Daniel Saad Nogueira Nunes, Felipe A. Louza, Simon Gog, Mauricio Ayala-Rincón, and Gonzalo Navarro. A grammar compression algorithm based on induced suffix sorting. In Ali Bilgin, Michael W. Marcellin, Joan Serra-Sagristà, and James A. Storer, editors, DCC 2018, pages 42-51. IEEE, 2018. URL: https://doi.org/10.1109/DCC.2018.00012.
  15. Carlos Ochoa and Gonzalo Navarro. Repair and all irreducible grammars are upper bounded by high-order empirical entropy. IEEE Transactions on Information Theory, 65(5):3160-3164, 2019. URL: https://doi.org/10.1109/TIT.2018.2871452.
  16. David Salomon. Data Compression: The Complete Reference. Springer London, 2007. URL: https://doi.org/10.1007/978-1-84628-603-2.
  17. Yasuo Tabei, Yoshimasa Takabatake, and Hiroshi Sakamoto. A succinct grammar compression. In Johannes Fischer and Peter Sanders, editors, Combinatorial Pattern Matching, pages 235-246, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. URL: https://doi.org/10.1007/978-3-642-38905-4_23.
  18. Akito Takasaka and Tomohiro I. Space-efficient SLP encoding for O(log N)-time random access. In String Processing and Information Retrieval (SPIRE 2024), volume 14899 of Lecture Notes in Computer Science, pages 336-347. Springer, 2024. URL: https://doi.org/10.1007/978-3-031-72200-4_25.
  19. Sebastiano Vigna. Broadword implementation of rank/select queries. In Catherine C. McGeoch, editor, WEA 2008, pages 154-168, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. URL: https://doi.org/10.1007/978-3-540-68552-4_12.
  20. Jia-Xing Yue, Jing Li, Louise Aigrain, Johan Hallin, Karl Persson, Karen Oliver, Anders Bergström, Paul Coupland, Jonas Warringer, Marco Cosentino Lagomarsino, Gilles Fischer, Richard Durbin, and Gianni Liti. Contrasting evolutionary genome dynamics between domesticated and wild yeasts. Nature Genetics, 49(6):913-924, April 2017. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact