Abstract 1 Introduction 2 Preliminaries 3 Collage Systems to Internal Collage Systems 4 MAX-SAT formulation to compute 𝒄^(𝑻) 5 Conclusions and future work References

On the Smallest Size of Internal Collage Systems

Soichiro Migita Kyushu Institute of Technology, Fukuoka, Japan    Kyotaro Uehata Kyushu Institute of Technology, Fukuoka, Japan    Tomohiro I ORCID Kyushu Institute of Technology, Fukuoka, Japan
Abstract

A Straight-Line Program (SLP) for a string T is a context-free grammar in Chomsky normal form that derives T only, which can be seen as a compressed form of T. Kida et al. introduced collage systems [Theor. Comput. Sci., 2003] to generalize SLPs by adding repetition rules and truncation rules. The smallest size c(T) of collage systems for T has gained attention to see how these generalized rules improve the compression ability of SLPs. Navarro et al. [IEEE Trans. Inf. Theory, 2021] showed that c(T)O(z(T)) and there is a string family with c(T)Ω(b(T)log|T|), where z(T) is the number of phrases in the Lempel-Ziv parsing of T and b(T) is the smallest size of bidirectional schemes for T. They also introduced a subclass of collage systems, called internal collage systems, and proved that its smallest size c^(T) for T is at least b(T). While c(T)c^(T) is obvious, it is unknown how large c^(T) is compared to c(T). In this paper, we prove that c^(T)=Θ(c(T)) by showing that any collage system of size m can be transformed into an internal collage system of size O(m) in O(m2) time. Thanks to this result, we can focus on internal collage systems to study the asymptotic behavior of c(T), which helps to suppress excess use of truncation rules. As a direct application, we get b(T)=O(c(T)), which answers an open question posed in [Navarro et al., IEEE Trans. Inf. Theory, 2021]. We also give a MAX-SAT formulation to compute c^(T) for a given T.

Keywords and phrases:
Collage Systems, Dictionary-based compression, Compressibility measures
Funding:
Tomohiro I: KAKENHI 24K02899, JST AIP Acceleration Research JPMJCR24U4
Copyright and License:
[Uncaptioned image] © Soichiro Migita, Kyotaro Uehata, and Tomohiro I; licensed under Creative Commons License CC-BY 4.0
2012 ACM Subject Classification:
Theory of computation Data compression
Related Version:
Previous Version: https://arxiv.org/abs/2509.11602
Editors:
Philip Bille and Nicola Prezza

1 Introduction

Grammar-based compression is a framework of lossless compression whose outcomes are modeled by grammars like Context-free grammars (CFGs). Grammar-based compression is popular because it is not only powerful to model the practical compressors such as LZ78 [17] and RePair [7], but also suitable to design algorithms working directly on compressed data [9]. In this paper, we focus on CFGs in Chomsky normal form that derive a single string, which are called Straight-Line Programs (SLPs) [3]. 111The definition of SLPs varies across the literature. For example, SLPs are not necessarily in Chomsky normal form in [9].

An SLP of size m for a string T has m nonterminals and the starting nonterminal is deterministically expanded to a unique string T according to production rules of the form Xa or XYZ, where X,Y,Z are nonterminals and a is a terminal symbol. The derived string from X, denoted by X, is X=a for the former rules and X=YZ for the latter rules called the concatenation rules.

Kida et al. [5] introduced collage systems to generalize SLPs by adding repetition rules and truncation rules. For a repetition rule (a.k.a. run-length rule) XYr with r>2, Xi is expanded to the r-times repeat of Y and we have X=Yr. For a truncation rule XY[b..e) with 1b<e|Y|+1, X is obtained by truncating Y to the substring Y[b..e), the substring of Y starting at b and ending at e1. 222In the original paper [5], the substring truncation is implemented by the prefix truncation and the suffix truncation. Collage systems with no truncation rules are also called Run-length SLPs (RLSLPs) [13] and collage systems with no repetition rules are also called composition systems [2]. The size of a collage system is measured by the number of its nonterminals and let c(T) denote the smallest size of collage systems for a string T.

Recently, compressibility measures for highly-repetitive strings have been extensively studied [10, 11]. Although computing c(T) for a given string T is NP-hard [4], it has gained attention as a compressibility measure to see how repetition and/or truncation rules improve the compression ability of SLPs. Navarro et al. [12] showed that c(T)O(z(T)) and there is a string family with c(T)Ω(b(T)log|T|), where z(T) is the size of LZ76 parsing [8] of T and b(T) is the smallest size of bidirectional schemes [16] for T. They also introduced a subclass of collage systems, called internal collage systems: An internal collage system must have production rules so that, for every nonterminal X, the starting nonterminal can be expanded to get a sequence that contains X without using truncation rules. This restriction helps to suppress excess use of truncation rules. In particular, a general collage system may have a nonterminal X such that X is not a substring of T but the substring of X is truncated and used to represent a part of T. This is not the case in internal collage systems where every nonterminal X can be reached from the starting nonterminal without truncation rules, implying that X appears in T at least once. Moreover, the restriction enables to define a parsing of T of size linear to the size of an internal collage system and prove its smallest size c^(T) is in Ω(b(T)) [12].

While c(T)c^(T) is obvious, it is unknown how large c^(T) is compared to c(T). In this paper, we prove that c^(T)=Θ(c(T)) by showing that any collage system of size m can be transformed into an internal collage system of size O(m) in O(m2) time. Thanks to this result, we can focus on internal collage systems to study the asymptotic behavior of c(T). As a direct application, we get b(T)=O(c(T)), which answers an open question posed in [12].

Kawamoto et al. [4] obtained a hardness result for computing c(T). We just remark that exactly the same proof works to get the following result for c^(T) because truncation rules are not used to solve the reduced problem, and therefore, there are no difference between c(T) and c^(T) in the reduction.

Theorem 1.

The problem of computing c^(T) for a given string T is NP-hard.

In this paper, we give a MAX-SAT formulation to compute c^(T) for a given T, extending the previous approaches for SLPs [1] and RLSLPs [4]. Since we now know that c^(T)=Θ(c(T)), this provides a tool to study the behavior of both c^(T) and c(T) through computing exact values of c^(T) for concrete strings (although we may be able to process only short strings in a realistic time).

2 Preliminaries

2.1 Basic notation

An integer interval {i,i+1,,j} is denoted by [i..j], where [i..j] represents the empty interval if i>j. Also, [i..j) denotes [i..j1].

Let Σ be an ordered finite alphabet. An element of Σ is called a string over Σ. The length of a string w is denoted by |w|. The empty string ε is the string of length 0, that is, |ε|=0. Let Σ+=Σ{ε} and Σk={wΣ|w|=k} for any non-negative integer k. The concatenation of two strings x and y is denoted by xy or simply xy. When a string w is represented by the concatenation of strings x, y and z (i.e., w=xyz), then x, y and z are called a prefix, substring, and suffix of w, respectively. A substring x of w is called proper if xw. For any string w and non-negative integer k, let wk denote the k-times repeat of w, i.e., w0=ε and wk=wk1w for any k>1.

A factorization of a string w is a sequence w1,w2,,wh of substrings of w such that w=w1w2wh. Each substring wi(1ih) is called a factor of the factorization.

The i-th symbol of a string w is denoted by w[i] for 1i|w|, and the substring of a string w that begins at position i and ends at position j is denoted by w[i..j] for 1ij|w|, i.e., w[i..j]=w[i]w[i+1]w[j]. For convenience, let w[i..j]=ε if j<i.

2.2 Collage Systems and Internal Collage Systems

A collage system [5] of size m has m nonterminals. Each nonterminal X derives a single string X that is defined by a unique rule to expand X of either one of the following forms, where Y and Z are nonterminals:

  • Xa is an atomic rule such that X=a, where aΣ.

  • XYZ is a concatenation rule such that X=YZ.

  • XYr with r>2 is a repetition rule such that X=Yr.

  • XY[b..e) with 1b<e|Y|+1 is a truncation rule such that X=Y[b..e).

We say that X or its rule refers to symbols in the righthand side of the rule. We assume that nonterminals are sorted such that every nonterminal refers to smaller nonterminals, which is always possible because a collage system must not have a reference loop to represent a finite single string. Also, we assume that there is no useless nonterminal, i.e., all nonterminals are involved in the process of expanding the starting nonterminal. A collage system is called internal if, for every nonterminal X, the starting nonterminal can be expanded to get a sequence that contains X without using truncation rules. Let c(T) (resp. c^(T)) denote the smallest size of collage systems (resp. internal collage systems) for a string T.

We define the binary parse tree of a collage system in a top-down manner as follows:

  • The root node is labeled with the starting nonterminal.

  • If a node is labeled with X that has atomic rule Xa, its only child is labeled with a.

  • If a node is labeled with X that has concatenation rule XYZ, its left child is labeled with Y and its right child is labeled with Z.

  • If a node is labeled with X that has repetition rule XYr, its left child is labeled with Y and its right child is labeled with Yr1.

  • If a node is labeled with X that has truncation rule XY[b..e), its only child is labeled with Y[b..e).

  • If a node does not fit in any of the cases above, it is a leaf.

Note that we treat Yr1, Y[b..e) and Y as different node labels, and a node with label Yr1 or Y[b..e) falls into the last case and always becomes a leaf in the binary parse tree. A collage system is internal if and only if every nonterminal appears as a node label in the binary parse tree.

See Figure 1 (resp. Figure 2) for examples of an internal collage system (resp. non-internal collage system) and their binary parse tree.

Figure 1: An illustration of the binary parse tree (right) for the internal collage system having six rules shown left. The internal nodes are depicted by circles, and the leaves by solid boxes. The characters derived from leaves are depicted with dotted boxes. This is an internal collage system because every nonterminal appears as a node label in the binary parse tree.
Figure 2: An illustration of the binary parse tree (right) for the non-internal collage system having six rules shown left. It is not an internal collage system because X4 does not appear as a node label in the binary parse tree.

Similarly to [12], we define the grammar tree of a collage system as the tree obtained from the binary parse tree by deleting all descendant nodes under any node that is not the leftmost node with the same label (see Figure 3 for an example).333The idea of grammar trees for internal collage systems is extended from partial parse trees for SLPs [14]. Since the grammar tree has every nonterminal at most once as the label of an internal node, we sometimes identify an internal node by its label.

Proposition 2.

The following statements hold for the grammar tree of an internal collage system of size m that contains σ atomic rules and m𝗍𝗋 truncation rules:

  • The number of internal nodes is m because every nonterminal appears “exactly once” as the label of an internal node in the grammar tree.

  • The number of leaves is mm𝗍𝗋σ+1 because every internal node for a concatenation or repetition rule has two children while every internal node for an atomic or truncation rule has a single child.

Figure 3: An illustration of the binary parse tree (right above) and the grammar tree (right below) for the internal collage system having 9 rules shown left. Three internal nodes with label X2, X3 and X6, turn into leaves in the grammar tree because they are not the leftmost nodes with their own label. Observe that Proposition 2 holds, i.e., the number of internal nodes is the size m=9 of the collage system, and the number of leaves is mm𝗍𝗋σ+1=912+1=7, where σ=2 is the alphabet size and m𝗍𝗋=1 is the number of truncation rules.

2.3 MAX-SAT

Given a set of clauses, the satisfiability (SAT) problem asks for an assignment of variables that satisfies all the clauses. An extension to SAT is MAX-SAT, where we are given hard clauses and soft clauses, and maximize the number of satisfying soft clauses while satisfying all the hard clauses.

3 Collage Systems to Internal Collage Systems

In this section, we show that there is an algorithm to convert a collage system of size m for a string T to an internal collage system of size O(m), and prove that c^(T)=Θ(c(T)).

Figure 4: Illustration for Case 1. The upper parts show where the truncated substring Q exists in X and the lower parts show how the converted collage system represents it without using X. The intervals in the truncation rules are abbreviated and shown as “[)”. The reference of a nonterminal Q is changed from an unreachable nonterminal X to a smaller nonterminal Y or Z. If the new reference is unreachable, it will be processed later.
Figure 5: Illustration for Case 2. At most four new nonterminals including V are enough to represent Q. Note that new rules that truncate Y are introduced, but Y is guaranteed to be reachable via Q.
Figure 6: Illustration for Case 3, where the set A (described in the proof of Theorem 3) is assumed to contain four truncation rules with lefthand sides Q1,Q2,Q3 and Q4. For each Qi(1i4), we introduce at most two new nonterminals, which are guaranteed to be reachable via Qi. All such nonterminals other than S and P refer to a reachable nonterminal S or P. The new truncation rule SY[) (resp. PZ[)) will be processed later if Y (resp. Z) is unreachable.
Theorem 3.

There is an algorithm to convert a collage system of size m for a string T to an internal collage system of size O(m) in O(m2) time.

Proof.

We call a nonterminal reachable if it appears as a node label of the grammar tree, and otherwise unreachable. Then, a collage system is internal if and only if every nonterminal is reachable. Our algorithm processes unreachable nonterminals X in a top-down manner and modifies the collage system to represent T without using X. When we get into an unreachable nonterminal X, all nonterminals above have been processed to be reachable, and therefore, X is referred to only by truncation rules because referring to Xi in a rule other than truncation rules implies that X is reachable. Hence, removing X from the collage system is equivalent to removing the truncation rules that refer to X. The process of removing these truncation rules depends on the following cases:

  • Case 0: When the truncation rule can be replaced with an atomic rule.

  • Case 1: When the truncation can be made from a single smaller nonterminal.

  • Case 2: When the truncation can be made from more than two repeating nonterminals.

  • Case 3: When the truncation can be made from two nonterminals.

Case 0.

Case 0 is the base case where Xa is an atomic rule. For any nonterminal Q that truncates X, it means that Q=a and it is obvious that the truncation rule can be replaced with Qa.

Case 1.

Suppose that we want to remove a truncation rule QX[b..e). Case 1 deals with the following situations:

  • When XYZ and the interval [b..e) is included in either one of [1..|Y|] or [|Y|+1..|Z|].

  • When XYr and the interval [b..e) is covered by a single Y, i.e., [b..e) is included in [(r1)|Y|+1..r|Y|] for some r[1..r].

  • When XY[b..e).

In either case, it is easy to replace the truncation rule QX[b..e) with a rule that truncates a smaller nonterminal Y or Z. If the smaller nonterminal is unreachable, the added truncation rule will be processed later when we get into the nonterminal.

Case 2.

Case 2 deals with a truncation rule QX[b..e) that truncates unreachable nonterminal X with repetition rule XYr spanning more than two Y’s. In this case, Q can be represented by uvw, where u is a (potentially empty) suffix of Y, v=Yr with r1 and w is a (potentially empty) prefix of Y. We add a new nonterminal U (resp. W) with a rule that truncates Y to represent u (resp. w) if u (resp. w) is not the empty string. We also add a new nonterminal V with repetition rule VYr if r>1.444If r=2, the added rule is categorized to concatenation rules in our definition. Finally, we can represent Q by concatenating at most three nonterminals for u, v and w using at most two concatenation rules. The process increases the size of the collage system by at most four because we create at most five rules and remove the original truncation rule QX[b..e). Note that Y is reachable because Q is reachable by inductive hypothesis and there is a path from Q to Y without truncation. Hence, added rules that truncate Y do not need to be processed later.

Case 3.

Case 3 deals with rules that truncate an unreachable nonterminal X with concatenation or repetition rule spanning exactly two nonterminals. We focus on the case with concatenation rule XYZ as the case with repetition rule can be processed similarly. Let A be the set of all rules that truncate X and their truncated intervals are not included in [1..|Y|] nor [|Y|+1..|X|]. Unlike other cases, we process all truncation rules in A at once.

Let s (resp. p) be the length of the longest suffix (resp. prefix) truncated from Y (resp. Z) by a truncation rule in A. We create a new nonterminal S that truncates the suffix of length s of Y and P that truncates the prefix of length p of Z. Then, any truncation made by a rule in A can be replaced with concatenation of a nonterminal that truncates S (or S itself) and a nonterminal that truncates P (or P itself). This increases the size of the collage system by at most two per truncation rule in A. Note that any nonterminal, including S and P, created during the process is reachable through the nonterminals in the lefthand sides of A, which are reachable by inductive hypothesis. Also, S and P are the only nonterminals that are created in the process and may truncate unreachable nonterminals, which will be processed later.

Correctness of conversion.

Each modification to the collage system does not change the string it represents. Since all nonterminals added during the process are reachable, the collage system becomes internal when we go through the process of removing unreachable nonterminals in the original collage system.

Size analysis.

The number of nonterminals in the collage system is increased by the process of removing a rule that truncates unreachable nonterminals by four in Case 2 and by two in Case 3. We have to take care of additional rules S and P created in Case 3 that may truncate unreachable nonterminals. The number of such additional truncation rules is bounded by 2(mm𝗍𝗋) in total, where m𝗍𝗋 is the number of truncation rules in the original collage system. Thus, the size of the resulting internal collage system is bounded by m+4(2(mm𝗍𝗋)+m𝗍𝗋)=9m4m𝗍𝗋9m.

Time complexity.

In every case, we can process a truncation rule in O(1) time each. While the truncation rule is completely removed in Case 0, Case 2 and Case 3, what we do in Case 1 is just switching its reference nonterminal to smaller nonterminals in O(1) time. This switching cost may add up to O(m) until we finally remove it in Case 0, Case 2 or Case 3. Since we remove O(m) truncation rules, the algorithm runs in O(m2) time.

It follows from Theorem 3 that c^(T)9c(T). Together with the fact that c(T)c^(T), we get the following theorem:

Theorem 4.

For any string T, it holds that c^(T)=Θ(c(T)).

4 MAX-SAT formulation to compute 𝒄^(𝑻)

In this section, we give a MAX-SAT formulation to compute c^(T) for a given string T. The formulation is based on the ICS-factorization T=F1Fh defined for an internal collage system such that Fi is the string derived from the i-th leaf of its grammar tree. Recall that h=mm𝗍𝗋σ+1 holds by Proposition 2, where m is the size and m𝗍𝗋 is the number of truncation rules of the internal collage system. Thus, minimizing the size m=h+m𝗍𝗋+σ1 is reduced to finding a valid ICS-factorization with the fewest factors and truncation rules.

Example 5.

The ICS-factorization of T=𝚋𝚊𝚋𝚋𝚋𝚊𝚋𝚋𝚊𝚊𝚋 for the internal collage system of Figure 3 has seven factors with F1=𝚋𝚊, F2=𝚋, F3=𝚋𝚋, F4=𝚊, F5=𝚋, F6=𝚋𝚊 and F7=𝚊𝚋.

The following Lemma gives a necessary and sufficient condition for a valid ICS-factorization, which extends Lemma 4.1 of [4] for RLSLPs to incorporate truncation rules.

Lemma 6.

Let T=F1Fh be a factorization of a string T and let si=1+k=1i1|Fk| for 1ih+1. The factorization is a valid ICS-factorization if and only if each factor Fk longer than 1 is categorized into either one of the types (A), (B) and (C) so that the following conditions hold:

  1. (i)

    Any factor Fk longer than 1 satisfies the following depending on its type:

    1. (A)

      Fk=T[sik..sjk+1) for some integers ikjk<k.

    2. (B)

      Fk=(T[sik..sjk+1))r1 for some integers ikjk=k1 and r3.

    3. (C)

      Fk is a substring of T[sik..sjk+1) for some integers ikjk with k<ik or k>jk.

  2. (ii)

    The set I=I1I2I3I4 of intervals is compatible with tree structures, where

    • I1={[sik..sjk+1)Fk is type-A},

    • I2={[sik..sjk+1)Fk is type-B},

    • I3={[sik..sk+1)Fk is type-B}, and

    • I4={[sik..sjk+1)Fk is type-C}.

  3. (iii)

    The interval for a Type-B factor is not in I.

  4. (iv)

    Each factor Fk can be given an integer Dk such that:

    • Dk=0 if |Fk|=1.

    • Dk>max{Dik,,Djk} if |Fk|>1.

Proof.

():

Suppose that T=F1Fh is the ICS-factorization of an internal collage system G. Let Li denote the i-th leaf of the grammar tree of G. We categorize a factor Fk longer than 1 and set integers ik and jk as follows.

  • If Lk is labeled with a nonterminal X, then Fk is type-A. Since Lk is not the leftmost node with label X, we can set integers ikjk<k so that Lik,Lik+1,,Ljk are the leaves under the internal node with label X to satisfy condition (i). The interval [sik..sjk+1) in I1 corresponds to the internal node X.

  • If Lk is the right child of a nonterminal X with a repetition rule XYr, then Fk is type-B. We can set integer ik so that Lik,Lik+1,,Lk1 are the leaves under the left child of X to satisfy condition (i). The interval [sik..sjk+1) in I2 corresponds to the left child of X, while the interval [sik..sk+1) in I3 corresponds to the internal node X.

  • If Lk is the child of a nonterminal X with a truncation rule XY[b..e), then Fk is type-C. We can set integers ik and jk so that Lik,Lik+1,,Ljk are the leaves under the internal node with label Y to satisfy condition (i). The interval [sik..sjk+1) in I4 corresponds to the internal node Y.

As seen above, every interval in I corresponds to a node of the grammar tree, and hence, it is compatible with tree structures. Also, the interval of a type-B factor corresponds to a leaf node labeled with Yr1, which implies that it is not in I. Finally, we prove condition (iv). For any factor Fk longer than 1, we have set ik and jk so that Fk is derived from a nonterminal that refers to smaller nonterminals deriving FikFjk. Since G does not have a reference loop of nonterminals, there is no reference loop of factors, too. Hence, we can assign integer Dk to Fk so that Dk is larger than integers {Dik,,Djk} assigned to factors it refers to. Assigning 0 to a factor of length 1, we see that condition (iv) holds.

():

Given that every factor Fk longer than 1 is categorized into types to satisfy conditions (i), (ii), (iii) and (iv). We can define a reference structure of factors according to condition (i). Referring to a type-B factor alone is avoided by condition (iii), and it is guaranteed by condition (iv) that there is no reference loop of factors. By condition (ii), I can identify the intervals corresponding to nodes of a tree, which are sufficient to define a reference structure of nonterminals. Although not all nodes are identified, we can fill missing internal nodes by adding concatenation rules to build an internal collage system for the ICS-factorization.

We modify the MAX-SAT formulation in [4] for RLSLPs to encode valid ICS-factorizations of smallest internal collage systems based on Lemma 6. We say that an ICS-factor longer than 1 refers to the substring T[sik..sjk+1) defined in Lemma 6 for each type. Let us call an integer Dk satisfying condition (iii) of Lemma 6 the reference depth of factor Fk or the characters covered by Fk. Note that a type-C factor may refer to a string to the right while type-A and type-B factors always refer to a substring to the left, which adds some complication to our formulation compared to the previous one for RLSLPs. We will encode reference depths to ensure that there is no reference loop.

In the next paragraph, we will define several Boolean variables with multiple parameters in subscripts. For any variable x with some missing subscripts specified by “”, let x denote the set of (possibly tuples of) feasible subscripts to fill the missing part(s). For example, 𝑟𝑒𝑓,A={(i,i,)i[1..n1],[2..ni+1],i[1..i],T[i..i+)=T[i..i+)}, and for some fixed i and , 𝑟𝑒𝑓i,A={i(i,i,)𝑟𝑒𝑓,A}. We also use “” to represent anonymous (arbitrary) subscripts, which are filtered out when used in x. For example, 𝑟𝑒𝑓,A={(i,)(i,i,)𝑟𝑒𝑓,A}.

For a given text T[1..n] of length n, we define Boolean variables as follows to encode Lemma 6.

  • fi, for i[1..n] and [1..n+1i]: fi,=1 iff T[i..i+) is an ICS-factor.

  • pi for i[1..n+1]: For in+1, pi=1 iff i is the starting position of an ICS-factor. pn+1 exists for technical reasons. We set p1=pn+1=1.

  • 𝑟𝑒𝑓ii,A for i[1..n1], [2..ni+1] and i[1..i] s.t. T[i..i+)=T[i..i+): 𝑟𝑒𝑓ii,A=1 iff T[i..i+) is a type-A factor that refers to T[i..i+).

  • 𝑟𝑒𝑓ii,B for i[1..n1], [2..ni+1] and i[i+1..i1] s.t. T[i..i+)=T[i..i+) and ii divides : 𝑟𝑒𝑓ii,B=1 iff T[i..i+) is a type-B factor that refers to T[i..i) to repeat it.

  • 𝑟𝑒𝑓i,i,C for i,i[1..n2], [2..ni+1], [2..ni+1] and [i..i+)[i..i+)= s.t. T[i..i+) is a substring of T[i..i+): 𝑟𝑒𝑓i,i,C=1 iff T[i..i+) is a type-C factor that refers to T[i..i+) to truncate it.

  • 𝑑𝑟𝑒𝑓i,i for (i,i,)𝑟𝑒𝑓,A with =, (i,i)𝑟𝑒𝑓,B with =ii, or (i,,i)𝑟𝑒𝑓,,C: 𝑑𝑟𝑒𝑓i,i=1 iff there is an ICS-factor that starts at i and refers to T[i..i+).

  • qi, for (i,)𝑑𝑟𝑒𝑓, or (i,i,)𝑟𝑒𝑓,B with =+ii: qi,=1 iff [i..i+) is in the set I defined in Lemma 6.

  • 𝑑𝑒𝑝𝑡ℎi,,d for i[1..n], [1..n+1i] and d[0..n]: 𝑑𝑒𝑝𝑡ℎi,,d=1 iff d is less than or equal to the maximum reference depth of characters in T[i..i+).

We next define constraints that the above variables must satisfy.

Since the factors and their starting positions must be consistent, we have:

(i,)f,:fi,pi(¬pi+1)(¬pi+1)pi+. (1)

If T[i..i+) with >1 cannot be a candidate of type-A, type-B nor type-C factor, fi, must be false, i.e.,

(i,)f,(𝑟𝑒𝑓,A𝑟𝑒𝑓,B𝑟𝑒𝑓,,C) with l>1:¬fi,l. (2)

If T[i..i+) is a type-A factor, there exists i𝑟𝑒𝑓i,A s.t. 𝑟𝑒𝑓ii,A is true. If T[i..i+) is a type-B factor, there exists i𝑟𝑒𝑓i,B s.t. 𝑟𝑒𝑓ii,B is true. If T[i..i+) is a type-C factor, there exists (i,)𝑟𝑒𝑓,i,C s.t. 𝑟𝑒𝑓i,i,C is true. Conversely, if one of the variables of the form 𝑟𝑒𝑓ii,A, 𝑟𝑒𝑓ii,B or 𝑟𝑒𝑓i,i,C is true, T[i..i+) must be a factor. This can be encoded as

(i,)𝑟𝑒𝑓,A𝑟𝑒𝑓,B𝑟𝑒𝑓,,C:
fi,(i𝑟𝑒𝑓i,A𝑟𝑒𝑓ii,A)(i𝑟𝑒𝑓i,B𝑟𝑒𝑓ii,B)((i,)𝑟𝑒𝑓,i,C𝑟𝑒𝑓i,i,C). (3)

For a fixed substring T[i..i+), at most one variable of the form 𝑟𝑒𝑓ii,A, 𝑟𝑒𝑓ii,B or 𝑟𝑒𝑓i,i,C is allowed to indicate that T[i..i+) is a factor. Thus, we require

(i,)𝑟𝑒𝑓,A𝑟𝑒𝑓,B𝑟𝑒𝑓,,C:
i𝑟𝑒𝑓i,A𝑟𝑒𝑓ii,A+i𝑟𝑒𝑓i,B𝑟𝑒𝑓ii,B+(i,)𝑟𝑒𝑓,i,C𝑟𝑒𝑓i,i,C1. (4)

Constraint 4 for a fixed (i,) can be encoded in size of |𝑟𝑒𝑓i,A𝑟𝑒𝑓i,B𝑟𝑒𝑓,i,C|=O(n2) using an efficient encoding for this kind of “at-most” constraints [15].

If 𝑑𝑟𝑒𝑓i,i is true, then there is a factor that refers to T[i..i+) and vice versa.

(i,,i)𝑑𝑟𝑒𝑓,:
𝑑𝑟𝑒𝑓i,i𝑟𝑒𝑓ii,A(𝑟𝑒𝑓ii,B𝑟𝑒𝑓ii,B)(𝑟𝑒𝑓i,i,C𝑟𝑒𝑓i,i,C). (5)

We let qi, summarize the information on whether T[i..i+) corresponds to a node implied by 𝑟𝑒𝑓,A, 𝑟𝑒𝑓,B and 𝑟𝑒𝑓,,C:

(i,)q,:qi,(i𝑑𝑟𝑒𝑓i,𝑑𝑟𝑒𝑓i,i)((i,)𝑟𝑒𝑓i,Bwith =+ii𝑟𝑒𝑓ii,B). (6)

If qi, is true, then i must be the starting position of an ICS-factor.

(i,)q,:qi,pi. (7)

If qi, is true, then T[i..i+) must not be a type-B factor:

(i,)q,,i′′𝑟𝑒𝑓i,B:qi,¬𝑟𝑒𝑓i′′i,B. (8)

The set of intervals indicated by q, must be compatible with tree structures. In other words, for any two substrings T[i1..i1+1) and T[i2..i2+2) with i1<i2<i1+1<i2+2, at most one of T[i1..i1+1) and T[i2..i2+2) can correspond to a node of the grammar tree. Thus, we require that

(i1,1),(i2,2)q, s.t. i1<i2<i1+1<i2+2:¬qi1,1¬qi2,2. (9)

Since the reference depth of a character is at least 0, 𝑑𝑒𝑝𝑡ℎi,1,0 is always true:

i[1..n]:𝑑𝑒𝑝𝑡ℎi,1,0 =1. (10)

If the reference depth of a character is at least d, then it is also at least d1:

i[1..n],d[1..n]:𝑑𝑒𝑝𝑡ℎi,1,d𝑑𝑒𝑝𝑡ℎi,1,d1. (11)

The reference depths of characters in the same factor are the same:

i[2..n],d[0..n]:¬pi𝑑𝑒𝑝𝑡ℎi,1,d=𝑑𝑒𝑝𝑡ℎi1,1,d. (12)

The reference depth of a substring is the maximum of the reference depths of characters in the substring:

i[1..n],[1..n+1i],d[0..n]:𝑑𝑒𝑝𝑡ℎi,,dj=ii+l1𝑑𝑒𝑝𝑡ℎj,1,d. (13)

If an ICS-factor starting at i refers to substring T[i..i+), the reference depth of T[i] is larger than the reference depth of T[i..i+), which is encoded as

(i,,i)𝑑𝑟𝑒𝑓,:𝑑𝑟𝑒𝑓i,id[1..n](¬𝑑𝑒𝑝𝑡ℎi,1,d¬𝑑𝑒𝑝𝑡ℎi,,d1). (14)

From the above SAT formulation, we get the following theorem:

Theorem 7.

There is a MAX-SAT formulation of size O(n4) to compute smallest internal collage systems for a string T of length n.

Proof.

Correctness of SAT formulation.

We prove the correctness of our SAT formulation presented in this section for ICS-factorizations. On the one hand, given an internal collage system, it is clear that a truth assignment to Boolean variables based on the definition will satisfy all constraints. On the other hand, given T and a truth assignment satisfying the constraints, we can deduce a factorization to satisfy the conditions (i), (ii), (iii) and (iv) of Lemma 6. First, the truth assignments of pi and Constraint (1) give a factorization of T where we regard T[i..i+) as a factor if and only if fi,=1. If fi,=1 with >1, the type of factor T[i..i+) and its reference to satisfy condition (i) are specified by the truth assignments of 𝑟𝑒𝑓,A, 𝑟𝑒𝑓,B and 𝑟𝑒𝑓,,C, which are suitably controlled by Constraints (2), (3) and (4). By Constraint (5), the information about the starting position of a factor and its reference is collected in variables 𝑑𝑟𝑒𝑓,. By Constraint (6), the truth assignments of q, give the set I of intervals of Lemma 6. The starting position of every interval of I must be the starting position of an ICS-factor, which is enforced by Constraint (7). Condition (iii) is fulfilled by Constraint (8). Constraint (9) ensures that I is compatible with tree structures to satisfy condition (ii). Finally, condition (iv) is ensured by Constraints (10), (11), (12), (13) and (14) using 𝑑𝑟𝑒𝑓, and 𝑑𝑒𝑝𝑡ℎ,,.

MAX-SAT formulation.

By Proposition 2, the size m of an internal collage system corresponding to an ICS-factorization is m=h+m𝗍𝗋+σ1, where h is the number of factors, m𝗍𝗋 is the number of truncation rules, and σ is the number of distinct characters in T. Since σ is fixed, m is minimized if h+m𝗍𝗋 is minimized. This can be encoded in MAX-SAT by introducing the set of soft clauses consisting of ¬p and ¬𝑟𝑒𝑓,,C because satisfying these soft clauses as many as possible leads to minimizing h+m𝗍𝗋.

Size analysis.

In total, we have O(n4) Boolean variables dominated by 𝑟𝑒𝑓,,C. The total size of the resulting MAX-SAT formulation is O(n4), dominated by Constraints (3), (4), (5), (6), (9), (13) and (14) where there are four free parameters each of which take O(n) different values.

5 Conclusions and future work

In this paper, we proposed an O(m2)-time algorithm to convert a collage system of size m for a string T to an internal collage system of size 9m. As a consequence, we obtained new bounds c^(T)=Θ(c(T)) and b(T)=O(c(T)) on the smallest size of collage systems. We also give a MAX-SAT formulation to compute c^(T) for a given T.

Future work includes the following directions:

  • Improve the current (probably loose) upper bound 9m for the size of converted internal collage system or give a nontrivial lower bound.

  • Improve O(m2) running time for the conversion algorithm.

  • Implement our MAX-SAT formulation and evaluate how well it scales compared to other formulations for SLPs [1] and RLSLPs [4].

  • Consider a formulation to compute c(T). Here, a new technique would be required because the current approach, inherited from [1, 4], depends on the fact that c^(T) is measured through the ICS-factorization of T, but it does not hold for general collage systems that have “external” references.

  • Prove or disprove that z(T)/c(T)O(1), which is left unknown in the relation between compressibility measures presented in [6, Fig. 1].

References

  • [1] Hideo Bannai, Keisuke Goto, Masakazu Ishihata, Shunsuke Kanda, Dominik Köppl, and Takaaki Nishimoto. Computing NP-hard repetitiveness measures via MAX-SAT. In Proc. 30th Annual European Symposium on Algorithms (ESA) 2022, volume 244 of LIPIcs, pages 12:1–12:16. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, 2022. doi:10.4230/LIPIcs.ESA.2022.12.
  • [2] Leszek Gasieniec, Marek Karpinski, Wojciech Plandowski, and Wojciech Rytter. Efficient algorithms for Lempel-Zip encoding (extended abstract). In Rolf G. Karlsson and Andrzej Lingas, editors, Proc. 5th Scandinavian Workshop on Algorithm Theory (SWAT) 1996, volume 1097 of Lecture Notes in Computer Science, pages 392–403. Springer, 1996. doi:10.1007/3-540-61422-2_148.
  • [3] Marek Karpinski, Wojciech Rytter, and Ayumi Shinohara. Pattern-matching for strings with short descriptions. In Proc. 6th Annual Symposium on Combinatorial Pattern Matching (CPM) 1995, pages 205–214, 1995. doi:10.1007/3-540-60044-2_44.
  • [4] Akiyoshi Kawamoto, Tomohiro I, Dominik Köppl, and Hideo Bannai. On the hardness of smallest RLSLPs and collage systems. In Proc. Data Compression Conference (DCC) 2024, pages 243–252. IEEE, 2024. doi:10.1109/DCC58796.2024.00032.
  • [5] Takuya Kida, Tetsuya Matsumoto, Yusuke Shibata, Masayuki Takeda, Ayumi Shinohara, and Setsuo Arikawa. Collage system: a unifying framework for compressed pattern matching. Theor. Comput. Sci., 298(1):253–272, 2003. doi:10.1016/S0304-3975(02)00426-7.
  • [6] Tomasz Kociumaka, Gonzalo Navarro, and Nicola Prezza. Toward a definitive compressibility measure for repetitive sequences. IEEE Trans. Inf. Theory, 69(4):2074–2092, 2023. doi:10.1109/TIT.2022.3224382.
  • [7] N. Jesper Larsson and Alistair Moffat. Offline dictionary-based compression. In Proc. Data Compression Conference (DCC) 1999, pages 296–305, 1999. doi:10.1109/DCC.1999.755679.
  • [8] Abraham Lempel and Jacob Ziv. On the complexity of finite sequences. IEEE Trans. Information Theory, 22(1):75–81, 1976. doi:10.1109/TIT.1976.1055501.
  • [9] Markus Lohrey. Algorithmics on slp-compressed strings: A survey. Groups Complexity Cryptology, 4(2):241–299, 2012. doi:10.1515/GCC-2012-0016.
  • [10] Gonzalo Navarro. Indexing highly repetitive string collections, part I: repetitiveness measures. ACM Comput. Surv., 54(2):29:1–29:31, 2021. doi:10.1145/3434399.
  • [11] Gonzalo Navarro. Indexing highly repetitive string collections, part II: compressed indexes. ACM Comput. Surv., 54(2):26:1–26:32, 2021. doi:10.1145/3432999.
  • [12] Gonzalo Navarro, Carlos Ochoa, and Nicola Prezza. On the approximation ratio of ordered parsings. IEEE Trans. Inf. Theory, 67(2):1008–1026, 2021. doi:10.1109/TIT.2020.3042746.
  • [13] Takaaki Nishimoto, Tomohiro I, Shunsuke Inenaga, Hideo Bannai, and Masayuki Takeda. Fully dynamic data structure for LCE queries in compressed space. In Proc. 41st International Symposium on Mathematical Foundations of Computer Science (MFCS) 2016, pages 72:1–72:15, 2016. doi:10.4230/LIPIcs.MFCS.2016.72.
  • [14] Wojciech Rytter. Application of Lempel-Ziv factorization to the approximation of grammar-based compression. Theor. Comput. Sci., 302(1-3):211–222, 2003. doi:10.1016/S0304-3975(02)00777-6.
  • [15] Carsten Sinz. Towards an optimal CNF encoding of boolean cardinality constraints. In Peter van Beek, editor, Proc. CP, volume 3709 of LNCS, pages 827–831. Springer, 2005. doi:10.1007/11564751_73.
  • [16] James A. Storer and Thomas G. Szymanski. Data compression via textural substitution. J. ACM, 29(4):928–951, 1982. doi:10.1145/322344.322346.
  • [17] Jacob Ziv and Abraham Lempel. Compression of individual sequences via variable-rate coding. IEEE Transactions on Information Theory, 24(5):530–536, 1978. doi:10.1109/TIT.1978.1055934.