Internal Dictionary Matching

We introduce data structures answering queries concerning the occurrences of patterns from a given dictionary D\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathsf {D}$$\end{document} in fragments of a given string T of length n. The dictionary is internal in the sense that each pattern in D\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathsf {D}$$\end{document} is given as a fragment of T. This way, D\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathsf {D}$$\end{document} takes space proportional to the number of patterns d=|D|\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$d=|\mathsf {D}|$$\end{document} rather than their total length, which could be Θ(n·d)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varTheta (n\cdot d)$$\end{document}. In particular, we consider the following types of queries: reporting and counting all occurrences of patterns from D\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathsf {D}$$\end{document} in a fragment T[i..j]\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T[i \mathinner {.\,.}j]$$\end{document} and reporting distinct patterns from D\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathsf {D}$$\end{document} that occur in T[i..j]\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$T[i \mathinner {.\,.}j]$$\end{document}. We show how to construct, in O((n+d)logO(1)n)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O((n+d) \log ^{O(1)} n)$$\end{document} time, a data structure that answers each of these queries in time O(logO(1)n+|output|)\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$O(\log ^{O(1)} n+| output |)$$\end{document}. The case of counting patterns is much more involved and needs a combination of a locally consistent parsing with orthogonal range searching. Reporting distinct patterns, on the other hand, uses the structure of maximal repetitions in strings. Finally, we provide tight—up to subpolynomial factors—upper and lower bounds for the case of a dynamic dictionary.


Introduction
In the problem of dictionary matching, which has been studied for more than forty years [1], we are given a dictionary D, consisting of d patterns, and the goal is to preprocess D so that presented with a text T we are able to efficiently compute the occurrences of the patterns from D in T .The Aho-Corasick automaton preprocesses the dictionary in linear time with respect to its total length and then processes T in time O(|T | + |output|) [1].Compressed indexes for dictionary matching [11], as well as indexes for approximate dictionary matching [16] have been studied.Dynamic dictionary matching in its more general version consists in the problem where a dynamic dictionary is maintained, text strings are presented as input and for each such text all the occurrences of patterns from the dictionary in the text have to be reported [2,3].
Internal queries in texts have received much attention in recent years.Among them, the Internal Pattern Matching (IPM) problem consists in preprocessing a text T of length n so that we can efficiently compute the occurrences of a substring of T in another substring of T .A nearly-linear sized data structure that allows for sublogarithmic-time IPM queries was presented in [30], while a linear sized data structure allowing for constant-time IPM queries in the case where the ratio between the lengths of the two substrings is constant was presented in [33].Other types of internal queries include computing the longest common prefix of two substrings of T , computing the periods of a substring of T , etc.We refer the interested reader to [31], which contains an overview of the literature.
We introduce the problem of Internal Dictionary Matching (IDM) that consists in answering the following types of queries for an internal dictionary D consisting of substrings of text T : given (i, j), report/count all occurrences of patterns from D in T [i . .j] and report the distinct patterns from D that occur in T [i . .j].
Let us formally define the problem and the types of queries that we consider.Let us consider Report(i, j) queries.One could answer them in O( j −i +|output|) time by running T [i . .j] over the Aho-Corasick automaton of D [1] or in time Õ(d + |output|) 1 by performing internal pattern matching [33] for each element of D individually.None of these approaches is satisfactory as they can require Ω(n) time in the worst case.The IDM problem in the special case of the dictionary D being the set of all palindromes in T has already been studied in [40], where authors proposed a data structure of size O(n log n) that returns the number of all distinct palindromes in T [i . .j] in O(log n) time.Let us note that in this case the total length of patterns might be quadratic, but the internal dictionary is of linear size and can be constructed in O(n) time [23].Our general solution can be applied, in particular, to other such dictionaries that satisfy these requirements, say, for the dictionary of all squares in T [7,18].

Our results and techniques
We focus on the case of a static dictionary, as it was defined above.We propose an Õ(n+d)-sized data structure, which can be built in time Õ(n+d) and answers all IDM queries in time Õ(1+|output|).The exact complexities, shown in Table 1, are achieved using slightly different techniques for different query types.The solutions for queries Exists(i, j) and Report(i, j), presented in Sect.3, are rather straightforward.The data structure for ReportDistinct(i, j) queries, described in Sect.4, heavily relies on the periodic structure of the input text and on tools that we borrow from computational geometry.An efficient solution for Count(i, j) queries, provided in Sect.5, is based on locally consistent parsing and further computational geometry tools.
Furthermore, in Sect.6, we consider a variant of the problem where the dictionary D is dynamic, i.e. allowing for interleaved IDM queries and updates to D (insertions/deletions of patterns).We first show a conditional lower bound for this problem: Unless the Online Boolean Matrix-Vector Multiplication conjecture [26] is false, the product of the time to process an update and the time to answer an Exists(i, j) query cannot be O(n 1− ) for any constant > 0. By building upon our solutions for static dictionaries, we also provide algorithms matching-up to subpolynomial factors-our conditional lower bound: For any 0 < α < 1, we show how to process updates in Õ(n α ) time and answer queries Exists(i, j), Report(i, j), ReportDistinct(i, j), and Count(i, j) in Õ(n 1−α + |output|) time.

Follow-up results
CountDistinct(i, j) is an equally interesting type of internal dictionary matching query, in which the number of distinct patterns from D that occur in T [i . .j] is to be returned.As has been already mentioned, a special case where D consists of all palindromes in T was considered in [40].In a very recent follow-up of our work by a superset of the authors [13], another special case where D consists of all squares in T was considered.For general dictionaries, the following results were shown in [13].

Preliminaries
We begin with basic definitions and notation generally following the conventions of [17].
The elements of Σ are called letters.By ε we denote an empty string.For two positions i and j on T , we denote by the fragment (sometimes called substring) of T that starts at position i and ends at position 1) space by specifying the indices i and j.A prefix of T is a fragment that starts at position 1 (T [1 . .j]) and a suffix is a fragment that ends at position n (T [i . . n], notation: T (i) ).We denote the reverse string of T by T R , i.e.
Let U be a string of length m with 0 < m ≤ n.We say that there exists an occurrence of U in T , or, more simply, that U occurs in T , when U is a fragment of T .We thus say that U occurs at the starting position i in T when If a string U is both a proper prefix and a proper suffix of a string A string T has a period p if and only if it has a border of length n − p.We refer to the smallest period as the period of the string, and denote it as per(T ), and, analogously, to the longest border as the border of the string.A string is called periodic if its period is no more than half of its length and aperiodic otherwise.
Consider a set of strings S. The trie of S is a rooted tree with edges labeled by single letters so that edges going down from a node have distinct labels.For each node v of the trie, the path-label of v is the concatenation of the edge labels on the path from the root to v, and the string-depth of v, denoted δ(v), is the length of the path-label.In the trie of S, the path-label of each node is a prefix of some string S ∈ S.Moreover, each prefix of a string S ∈ S is the path-label of a unique node of the trie, called its locus.In the compact trie of S, only branching nodes (with at least two children) and terminal nodes (loci of strings S ∈ S) are stored explicitly.The remaining implicit nodes of the trie (non-terminal nodes with exactly one child) are dissolved so that each edge of the compact trie can be viewed as an upward maximal path of implicit nodes starting with an explicit node.Subsequent nodes on this path are indexed by consecutive integers so that the index of the starting explicit node is zero.Note that each node belongs to a unique path of that kind.Thus, each node of the trie can be represented in the compact trie by the edge it belongs to and an index within the corresponding path.
The suffix tree T(T ) of a string T of length n is the compact trie of {T (i) $ : 1 ≤ i ≤ n}, where $ / ∈ Σ is a sentinel character that is lexicographically smaller than all the letters in Σ; this ensures that all terminal nodes are leaves.The suffix tree of a string of length n over an integer ordered alphabet can be computed in time and space O(n) [19].Once the suffix tree is constructed, it can be traversed in a depth-first manner to compute the string-depth δ(v) for each explicit node v.For each 1 ≤ i ≤ n, the terminal node with path-label T (i) $ is additionally labelled with index i.
We say that a tree is a weighted tree if it is a rooted tree with an integer weight on each node v, denoted by ω(v), such that the weight of the root is zero and ω(u) < ω(v) if u is the parent of v.We say that a node v is a weighted ancestor at depth of a node u if v is the highest ancestor of u with weight of at least .After O(n)-time preprocessing, weighted ancestor queries for nodes of a weighted tree T of size n with integer weights from a universe [1 . .U ] can be answered in O(log log U ) time per query [4].Moreover, any batch of k weighted ancestor queries can be answered in O(n + k) time [32,Section 7].If ω has a property that the difference of weights of a child and its parent is always equal to 1, then the queries can be answered in O(1) time after O(n)-time preprocessing [8]; in this special case the values ω are called levels and the queries are called level ancestor queries.The suffix tree T(T ) is a weighted tree with ω(v) = δ(v) for all v. Hence, the locus of a fragment T [i . .j] in T(T ) is the weighted ancestor of the terminal node with path-label T (i) $ at string-depth j − i + 1.
The elements of the dictionary D are called patterns.Henceforth we assume that ε / ∈ D, i.e. the length of each P ∈ D is at least 1.If ε was in D, we could trivially treat it individually.We further assume that each pattern of D is given by the starting and ending positions of its occurrence in T .Thus, the size of the dictionary d = |D| refers to the number of strings in D and not their total length.

EXISTS(i, j) and REPORT(i, j) Queries
We first present a convenient modification to the suffix tree with respect to a dictionary D; see Fig. 2. Definition 3.1 A D-modified suffix tree of a string T is a tree with leaves corresponding to non-empty suffixes of T $ and internal nodes corresponding to {ε} ∪ D. A node corresponding to string U is an ancestor of a node corresponding to string V if and only if U is a prefix of V .Each node stores its level as well as its string-depth δ.Moreover, each leaf node stores its label (i.e., the node corresponding to T (i) $ stores the label i).Proof The D-modified suffix tree is obtained from the suffix tree T(T ) in two steps.In the first step, we mark all nodes of T(T ) with path-label equal to a pattern P ∈ D∪{ε}: if any of them are implicit, we also make them explicit; see the annotated nodes in Fig. 2 (top).Note that we can find the loci of the patterns in T(T ) in O(n + d) time by answering the weighted ancestor queries as a batch [32,Section 7].We are left with the problem of making possibly several implicit nodes explicit on each edge of the tree.For this, the implicit nodes on each edge need to be sorted by their string-depths.We can avoid the local sorting, which could add ω(d) to the complexity, if we sort globally in time O(n + d) using bucket sort.
In the second step, we traverse the tree top-down and compute, for each marked node or leaf u, its nearest marked ancestor v = u; node v becomes the parent of node u in the D-modified suffix tree.The string-depths of nodes from the original suffix tree are retained.
We state the following simple lemma.

Lemma 3.3 With the D-modified suffix tree of T at hand and O(n + d) additional
space, given positions a, j in T with a ≤ j, we can compute all P ∈ D that occur as prefixes of T [a . .j] in time O (1 + |output|).
Proof We start from the root of the D-modified suffix tree and go down towards the leaf with label a.We report the path-labels of all encountered nodes v as long as δ(v) ≤ j − a + 1 is satisfied.We stop when this inequality is not satisfied.
The path is traversed using level ancestor queries on the D-modified suffix tree asked from the leaf [8].
The D-modified suffix tree enables us to efficiently answer Exists(i, j) and Report(i, j) queries.Let us introduce two auxiliary length-n arrays that can be computed from the D-modified suffix tree; the first one will be used in the solution in this section, whereas the other one in the following sections: 4

REPORTDISTINCT(i, j) Queries
Below, we present an algorithm that reports patterns from D occurring in T [i . .j], allowing for O(1) copies of each pattern on the output.We can then sort these patterns, remove duplicates, and report distinct ones using an additional global array of counters, one for each pattern.
Let us first partition D into D 0 , . . ., D log n such that D k = {P ∈ D : log |P| = k}.We call D k a k-dictionary.We now show how to process a single k-dictionary D k ; the query procedure may clearly assume k ≤ log |T [i . .j]|.
We first build the D k -modified suffix tree of T .Then, we compute the array L k := Longest(D k ), assign to all the patterns of D k equal to some T [a . .L k [a]] integer identifiers id (or colors) in [1 . .n], and construct an array I k [a] = id(P), where P = T [a . .L k [a]].Technically, in order to compute the array I k efficiently, for each index a, in addition to L k [a], we store the node of the D k -modified suffix tree representing T [a . .L k [a]]; this node is obtained as in the Proof of Lemma 3.5.
We then construct the data structure specified in the following theorem for I k ; this data structure, due to Muthukrishnan [38], allows for efficient colored range reporting queries.

Observation 4.2 Any pattern of a k-dictionary D k occurring in T at position p ∈
Based on this observation, we report the remaining patterns that start in [i . .t) using the D k -modified suffix tree, in O(1 + |output|) time, following parent pointers and temporarily marking the loci of reported patterns to avoid double-reporting.We have shown the following lemma.

Lemma 4.3 All distinct patterns from dictionary
We thus now only have to compute the patterns from D k that occur in T [t . .j].To this end, we further partition D k k > 1 to a periodic k-dictionary and an aperiodic k-dictionary: Note that we can partition D k in O(|D k |) time using the so-called 2-period queries of [6,31,33].Such a query decides whether a given fragment of the text is periodic and, if so, it also returns its period.It can be answered in O(1) time after an O(n)-time preprocessing of the text.
Let us provide some intuition behind this partition of D k .As shown next, each pattern in D aper k can occur only a constant number of times in T [t . .j]. (If two occurrences of a pattern have a "large" overlap, then this pattern must have a "small" period.)Patterns in D per k may have many occurrences; however, as we will show below, all of them occur in one of a constant number of runs (formally defined in Sect.4.2).Our algorithm processes each such run R, efficiently computing a single occurrence of each P ∈ D per k that occurs in R.

Processing an Aperiodic k-dictionary
We make use of the following sparsity property.

Processing a Periodic k-dictionary
Our solution for periodic patterns relies on the well-studied theory of runs (maximal repetitions) in strings.A run is a periodic fragment R = T [a . .b] which can be extended neither to the left nor to the right without increasing the period p = per(R), that is, provided that the respective positions exist.The number of runs in a string of length n is O(n) and all the runs can be computed in O(n) time [6,34].The following property of runs is known [31,33].Observation 4. 6 Let P be a periodic pattern.Every occurrence of P in T is contained within a unique run R in T with per(R) = per(P).We say that this run R extends the occurrence.
Let R be the set of all runs in T .We construct for all k ∈ [0 . .log n ] the sets of runs Note that these sets might not be disjoint; however,

below) and thus their total size is O(n).
Example 4. 7 For the string T in Fig. 4, we have that is, runs from R k whose overlap with the fragment U contains at least 2 k positions.The following lemma is known, but we include a proof for the sake of completeness and to help the reader to develop some intuition.
Proof Since each of the runs in R k is of length at least 2 k , it suffices to prove that any position i of T is contained in at most 3 runs from R k .
Let m = 2 k 3 .Let us suppose for the sake of contradiction that some position i is contained in at least four runs from R k .Observe that each of the runs must cover all positions of at least one of the fragments . By the pigeonhole principle, two of these runs, say R 1 and R 2 , have an overlap of at least 2m positions.Further, we have This contradicts the following known fact, being a consequence of the periodicity lemma [20]: If R 1 and R 2 are two different overlapping runs with periods per(R 1 ) and per(R 2 ), then Strategy Let us now provide a high-level description of our approach for processing D per k .Given a fragment U = T [t . .j], we will first identify the set R k (U ) of runs in R k that have a sufficient overlap with U .There is a constant number of them by Lemma 4.8.For an occurrence of a pattern P ∈ D per k in U , the unique run R extending this occurrence of P must be in R k (U ).We will preprocess each run R ∈ R k (U ) in order to be able to compute a unique (the leftmost) occurrence of each such pattern P in R ∩ U .
Let us now focus on the structure of the problem that allows us to only compute such a unique occurrence.A string S is called primitive if it cannot be expressed as X k for a string X and an integer k > 1.The primitivity lemma states that a non-empty string V is primitive if and only if it occurs only twice in V V : as a prefix and as a suffix.Moreover, any |per(U )|-length fragment of a string U is primitive [17].The following consequence of these two standard facts will be used for P ∈ D per k and Q being (a fragment of) a run that extends its occurrence.Proof First, let us prove that P has at least one occurrence in the first q := per(Q) positions of Q. Suppose, for the sake of contradiction, that the leftmost occurrence of P in Q is at some position j > q.Then, by periodicity, we have Q We now proceed to showing that P has exactly one occurrence in the first q positions of Q. Towards a contradiction, suppose that this is not the case and that P occurs at positions i, j ∈ [1 . .q], with i < j.Now, let us consider neither as a prefix nor as a suffix.Hence, V is not primitive, a contradiction.
We next present our data structure.
Construction of the data structure As discussed above, we compute R k in O(n) time.We also build the D   ReportDistinct(i, j) queries in O(log n + |output|) time.In the next subsection, we carefully reduce the space occupied by our data structure.

Reducing the Space Usage
The space occupied by our data structure can be reduced to O(n + d).Let us notice that the O(n log n) term in the space complexity comes from several data structures that take O(n + d ) or O(n) space for a given dictionary D and are stored in our solution for some or all of the dictionaries D k , D The only additional operation we now need to support is determining the parent of a given leaf in the original D -modified suffix tree (before the leaves were trimmed).This can be done using the nearest colored ancestor data structure of [22] over the D-modified suffix tree.In such queries, given a tree with colored nodes, we want to be able to compute the nearest ancestor of a query node v that has a color c specified in the query.For a tree of size N , the data structure of [22] achieves O(log log N ) time per query after O(N )-time preprocessing.We can, however, exploit the fact that we only have palette = O(log n) colors to obtain constant-time queries within the same construction time.
It is shown in [22] that, in order to answer nearest colored ancestor queries in a tree with N nodes, it is enough to store some arrays of total size O(N ) and predecessor data structures for O(palette) subsets of [1 . .2N ] whose total size is O(N ).Here, a predecessor query pred S (i), for a set S and number i, returns the maximum element of S that does not exceed i.The time needed to compute the sets for the predecessor data structures and the arrays is O(N ).The time complexity of the query is proportional to the time required for answering a constant number of predecessor queries over the aforementioned sets.We implement a predecessor data structure for a set S ⊆ [1 . .2N ] using O(N ) bits of space as follows.We store a bitmap that has the ith bit set if and only if i ∈ S and augment it with a data structure that answers rank and select queries in O(1) time and requires o(N ) additional bits of space [15,27].Such a component can be constructed in O(N / log N ) time [5,37].Note that pred S (i) = select(rank(i)).We thus use O((n + d) log n) bits, i.e., O(n + d) machine words, in total.

Part (b)
The arrays Longest(D ) and Shortest(D ) do not need to be stored explicitly since their elements can be retrieved using the parent pointers and O(1)-time level ancestor queries in the D -modified suffix tree (Lemma 3.5).
It was shown in [21] that an O(1)-query-time RMQ data structure for an array of size N can be implemented using O(N ) bits.This data structure only returns the index of the minimum value in the given range.To construct the data structures, we build the underlying arrays one by one and store the relevant RMQ data structures, which require O(n log n) bits, i.e.O(n) words, in total.
Part (c) For colored range reporting over an array A, the main component of the data structure underlying Theorem 4.1 from [38] is an RMQ data structure over an array

COUNT(i, j) Queries
We first solve an auxiliary Breakpoints-Anchor IPM problem and apply it to answer Count(i, j) queries in O(log 2 n/ log log n) time with a data structure of size O(nd).Then, we employ a locally consistent parsing to reduce the space usage to O(n + d log n).

An Auxiliary Problem
For a string S, by inter-position i + 1/2 we refer to a location between positions i and i + 1 in S. We also refer to inter-positions 1/2 and |S| + 1/2.
We consider a variant of the Internal Pattern Matching (IPM) problem, in which we are given a set of inter-positions (breakpoints) B of P and upon query we are to compute all fragments of T [i . .j] that match P and align a specific inter-position (anchor) β of the text with some inter-position in B. Before we proceed with the solution to the auxiliary problem, let us recall known results on 2D counting problems.In the 2D orthogonal range counting problem, one is to preprocess an n × n grid with O(n) marked points so that upon query [x 1 , y 1 ] × [x 2 , y 2 ], the number of points in this rectangle can be computed efficiently.In the (dual) 2D range stabbing counting problem, one is to preprocess the grid with O(n) rectangles so that upon query (x, y) the number of (stabbed) rectangles that contain (x, y) can be retrieved efficiently.The counting version of range stabbing queries in 2D reduces to two-sided range counting queries in 2D as follows (cf.[39]).For each rectangle [x 1 , y 1 ] × [x 2 , y 2 ] in grid G, we add points (x 1 , y 1 ) and (x 2 + 1, y 2 + 1) with weight 1 and points (x 1 , y 2 + 1) and (x 2 , y 1 + 1) with weight −1 in a grid G .Then the number of rectangles stabbed by point (a, b) in G is equal to the sum of weights of points in (−∞, a] × (−∞, b] in G .We will use the following result in our solution to Breakpoints-Anchor IPM (Lemma 5.4).

Theorem 5.1 ([12]) Range counting queries for n points in 2D (rank space) can be answered in time O(log n/ log log n) with a data structure of size O(n) that can be constructed in time O(n
. m] : b ∈ B}∪{ε} and consider the set W 2 obtained by adding U $ and U # for each element U of W 1 to an initially empty set, where $ is a letter smaller (resp.# is larger) than all the letters in Σ.Let W be the compact trie for the set of strings W 2 ; the trie W can be constructed efficiently using a rather standard approach.

Lemma 5.2 W can be constructed in O(n + |B|) time.
Proof First we sort all elements of W 2 lexicographically.For this, (implicit or explicit) nodes of T(T ) that correspond to strings in W 1 are marked using batch weighted ancestor queries.Implicit nodes on each edge are sorted top-down, off-line for all edges of T(T ), using radix sort.Then we traverse T(T ) using left-to-right depth first search.When a node corresponding to a string U ∈ W 1 is visited for the first/last time, we add U $/U # to the sorted list of W 2 , which we further denote by L. We can compute longest common prefixes of fragments of T in O(1) time after O(n)-time preprocessing.This lets us compute longest common prefixes of all pairs of strings that are adjacent in L in O(n + |B|) total time.Then the construction of W mimics the algorithm for constructing the suffix tree of a string from its suffix and LCP arrays; see e.g.[17].
We also build the W 1 -modified suffix tree of T and preprocess it for weighted ancestor queries.Moreover, for each string U ∈ W 1 , we store pointers to the leaves of W representing U $ and U #.
in the grid.An illustration is provided in Fig. 6.We then preprocess the grid for the counting version of 2D range stabbing queries, employing Theorem 5.1.
Query Without loss of generality, we may assume i − 1 2 ≤ β ≤ j + 1 2 ; otherwise Count β (i, j) = 0. Let U be the longest string in W 1 that is a prefix of T [ β . .j]; this string can be determined in O(log log n) time using a weighted ancestor query in the W 1 -modified suffix tree of T .Moreover, let u be the leaf of W representing U $ (accessed through a pointer stored at U ).We execute a symmetric procedure to find the longest string V ∈ Z 1 that is a prefix of (T [i . .β ]) R and determine the leaf v of W R representing V $.The following lemma is obtained by building a trie W for the union of the sets W 2 defined in the above proof for each pattern (similarly for W R ) and adding all rectangles to a single grid.A warm-up solution for Count(i, j) Naively, Lemma 5.5 can be applied as follows to answer Count(i, j) queries.Let us set B P = {p + 1/2 : p ∈ [1 . .|P| − 1]} for each pattern P ∈ D and construct the data structure of Lemma 5.5.We build a balanced binary tree BT on top of the text and for each node v in BT define val(v) to be the fragment consisting of the characters corresponding to the leaves in the subtree of v.Note that if v is a leaf, then |val(v)| = 1; otherwise, val(v) = val(u )val(u r ), where u and u r are the children of v.For each node v in BT, we precompute and store the count for val(v), defined as the number of occurrences of patterns from D in val(v).If v is a leaf, this count equals |{P ∈ D : P = val(v)}| and can be determined easily due to |val(v)| = 1.Otherwise, each occurrence is contained in val(u ), is contained in val(u r ), or spans both val(u ) and val(u r ).Hence, we sum the answers for the children u and u r of v and add the result of a Breakpoints-Anchor IDM query in val(v) with the anchor between val(u ) and val(u r ).
To answer a query concerning T [i . .j], we recursively count the occurrences in the intersection of val(v) with T [i . .j], starting from the root r of BT as it satisfies val(r If the intersection is empty, the result is 0, and if val(v) is contained in T [i . .j], we can use the precomputed count.Otherwise, we recurse on the children u and u r of v and sum the resulting counts.It remains to add the number of occurrences spanning across both val(u ) and val(u r ).This value is non-zero only if T [i . .j] spans both these fragments, and it can be determined from a Breakpoints-Anchor IDM query in the intersection of val(v) and T [i . .j] with the anchor between val(u ) and val(u r ).
The query time is O(log 2 n/ log log n) since non-trivial recursive calls are made only for nodes on the paths from the root r to the leaves representing T [i] and T [ j].Nevertheless, the space required for this "solution" can be Ω(nd), which is unacceptable.Below, we refine this technique using a locally consistent parsing; our goal is to decrease the size of each set B P from Θ(|P|) to O(log n).

Recompression
A run-length straight line program (RLSLP) is a context-free grammar which generates exactly one string and contains two kinds of non-terminal symbols: concatenations with production of the form A → BC (for symbols B, C) and powers with production of the form A → B k (for a symbol B and an integer k ≥ 2).Every symbol A generates a unique non-empty string denoted g(A).
Each symbol A is also associated with its parse tree PT(A) consisting of a root with a label A to which zero or more subtrees are attached: if A is a terminal, there are no subtrees; if A → BC is a concatenation symbol, then PT(B) and PT(C) are attached; if A → B k is a power symbol, then k copies of PT(B) are attached.Note that if we traverse the leaves of PT(A) from left to right, spelling out the corresponding non-terminals, then we obtain g(A).The parse tree PT of the whole RLSLP generating a string T is defined as PT(S) for the starting symbol S. We define the value val(v) of a node v in PT to be the fragment T [a . .b] corresponding to the leaves T [a], . . ., T [b] in the subtree of v.Note that val(v) is an occurrence of g(A), where A is the label of v.A sequence of nodes in PT is a chain if their values are consecutive fragments in T .
The recompression technique by Jeż [28,29] consists in the construction of a particular RLSLP generating the input text T .The underlying parse tree PT is of depth O(log n) and it can be constructed in O(n) time.As observed by I [41], this parse tree PT is locally consistent in a certain sense.To formalize this property, he introduced the popped sequence of every fragment T Let t be the run-length encoding of the popped sequence of a substring S of T .If |S| = 1, then we set L(S) = ∅; otherwise, we define By Theorem 5.6, the set L(S) can be constructed in O(log n) time given an arbitrary occurrence of S in T .

Lemma 5.7 Let v be a non-leaf node of PT and let T [a . . b] be an occurrence of S contained in val(v), but not contained in val(u) for any child u of
Proof Consider the chain v 1 , . . ., v p of nodes in PT whose values constitute T [a . .b] and whose labels form the popped sequence of S.These nodes are descendants of v and, since |S| > 1 guarantees p > 1, they are proper descendants of v (i.e., descendants of children of v).Consequently, T [a . .c] = val(v 1 ) • • • val(v q ), where v 1 , . . ., v q is the longest prefix of v 1 , . . ., v p consisting of descendants of the same child of v.
If the labels of v q and v q+1 are distinct, then their labels belong to distinct runs in the popped sequence, and thus |T [a . .c]| ∈ L(U ).(See Fig. 7 for an illustration of this case.) Otherwise, Theorem 5.6 guarantees that v q and v q+1 share the same parent.As they are descendants of different children of v, their parent must be v.Due to this, and since v 1 , . . ., v q form a chain consisting of descendants of the same child of v, we have q The proof of the second claim is symmetric.
Data structure We use recompression to build an RLSLP generating T and the underlying parse tree PT.We also construct the data structure for Breakpoints-Anchor IDM queries of Lemma 5.5 with breakpoints B P = {i + 1 2 : i ∈ L(P)} for each pattern P ∈ D.Moreover, for every symbol A we store the number of occurrences of patterns from D in g(A).Additionally, if A → B k is a power, we also store the number of occurrences in g(B i ) for i ∈  Next, we process the RLSLP in a bottom-up fashion.If A is a terminal, its count is easily determined.If A → BC is a concatenation, we sum the counts for B and C and the number of occurrences spanning both g(B) and g(C).To determine the latter value, we fix an arbitrary node v with label A and denote its children u , u r .By Lemma 5.7, any occurrence of P intersecting both val(u ) and val(u r ) has a breakpoint aligned to the inter-position between the two fragments.Hence, the third summand is the result of a Breakpoints-Anchor IDM query in val(v) with the anchor between val(u ) and val(u r ).Finally, if A → B k , then to determine the count in g(B i ), we add the count for B, the count in g(B i−1 ), and the number of occurrences in B i spanning both the prefix B and the suffix B i−1 .To find the latter value, we fix an arbitrary node v with label A, denote its children u 1 , . . ., u k , and make a Breakpoints-Anchor IDM query in val(u 1 ) • • • val(u i ) with the anchor between val(u 1 ) and val(u 2 ).The correctness of this step follows from Lemma 5.7.The running time of this processing is O(n log n/ log log n), so the overall construction time is O(n log n/ log log n + d log 3/2 n).
Query Upon a query Count(i, j), we proceed essentially as in the warm-up solution: we recursively count the occurrences contained in the intersection of T [i . .j] with val(v) for nodes v in PT, starting from the root of PT.If the two fragments are disjoint, the result is 0, and if val(v) is contained in T [i . .j], it is the count precomputed for the label of v. Otherwise, the label of v is a non-terminal.If it is a concatenation symbol, we recurse on both children u , u r of v and sum the obtained counts.If T [i . .j] spans both val(u ) and val(u r ), we also add the result of a Breakpoints-Anchor IDM query in the intersection of T [i . .j] with val(v) and the anchor between val(u ) and val(u r ).If the label is a power symbol A → B k , we determine which of the children u 1 , . . ., u k of v are spanned by T [i . .j].We denote these children by u , . . ., u r and recurse on u and on u r .If r > , we also make a Breakpoints-Anchor IDM query in the intersection of T [i . .j] with val(u ) • • • val(u r ) and anchor between val(u ) and val(u +1 ).If r > + 1, we further add the precomputed value for g(B r − −1 ) to account for the occurrences contained in val(u +1 ) • • • val(u r −1 ) and make a Breakpoints-Anchor IDM query in the intersection of T [i . .j] with val(u +1 ) • • • val(u r ) and anchor between u r −1 and u r .By Lemma 5.7, the answer is the sum of the up to five values computed.The overall query time is O(log 2 n/ log log n) since we make O(log n) non-trivial recursive calls and each of them is processed in O(log n/ log log n) time.
We arrive at the main result of this section.

Dynamic Dictionaries
In this section, we study the IDM problem in the case that insertions to and deletions from the dictionary are allowed.First, we present a conditional lower bound for this problem that is based on the conjectured hardness of a known problem in the online setting.In the remainder, we focus on providing algorithms matching this conditional lower bound.

Conditional Lower Bound
In the Online Boolean Matrix-Vector Multiplication (OMv) problem, we are given as input an n × n boolean matrix M.Then, we are given in an online fashion a sequence of n vectors r 1 , . . ., r n , each of size n.For each such vector r i , we are required to output Mr i before receiving r i+1 .
We now present a restricted version of [26, Theorem 2.2] which is sufficient for our purposes.Theorem 6.2 ([26]) For all constants γ, > 0, the OMv conjecture implies that there is no algorithm that given as input an r 1 × r 2 matrix M, with r 1 = r γ 2 , preprocesses M in time polynomial in r 1 +r 2 , and then, presented with a vector v of size r 2 , computes Mv in time O(r 1+γ − 2 ) with error probability at most 1/3.We proceed to obtain an OMv-based conditional lower bound for IDM in the case of a dynamic dictionary.Our lower bound is stated for Exists(i, j) queries, but it clearly carries over to the remaining query types considered in this work.Theorem 6.3For all constants α, β > 0 with α + β < 1, the OMv conjecture implies that there is no algorithm that preprocesses T and D in time polynomial in n, performs insertions to D in time O(n α ), and answers Exists(i, j) queries in time O(n β ) in an online manner with error probability at most 1/3.Proof Let us suppose that there is such an algorithm and set γ = (α + /2)/(β + /2), where = 1 − α − β.Given an r 1 × r 2 matrix M satisfying r 1 = r γ 2 , we construct a text T of length n = r 1 r 2 + r 2 as follows.Let T be a text created by concatenating the rows of M in increasing order.Then T is obtained by assigning to each non-zero element of T the column index of the matrix entry it originates from, and appending one by one the integers in [1 . .r 2 ] in increasing order.Formally, for i ∈ (We append these letters in order to ensure that the dictionary that we construct below is internal.) We compute Mv as follows.We add the indices of v's non-zero entries into an initially empty dictionary.We then perform queries Exists(1 + (t − 1)r 2 , tr 2 ) for t ∈ 1..r 1 .The answer to query the Exists(1+(t −1)r 2 , tr 2 , ) is equal to the boolean dot product of the tth row of M with v.We thus obtain Mv, with each entry correct with probability at least 2/3.We can guarantee that the whole vector Mv is correct with probability at least 1 − n −Ω(1) ≥ 2/3 by maintaining Θ(log n) independent instances of the algorithm and taking the dominant answer to each Exists query.In total, we perform Õ(r 2 ) insertions to D and Õ(r 1 ) Exists queries.Thus, the total time required is Õ(r ) for > 0. Conjecture 6.1 would be disproved due to Theorem 6.2.

Remark 6.5
The proof of the above theorem requires a large alphabet.Let us describe a straightforward modification to this proof that yields the same lower bound for ternary alphabets.We add a $ between every pair of consecutive letters in T and then replace each integer in T with its binary representation.Similarly, the patterns we add to or remove from D are the same integers as above, but represented in binary.

Upper Bound
Before we proceed solving the dynamic IDM problem, let us recall known results for internal pattern matching queries.In this setting, each query specifies a fragment T [i . .j] and a substring P of the text and asks questions analogous to those that we have defined for internal dictionary matching.Note that we assume that the pattern P is a substring of T and is given by one of its occurrences.We answer Count(P, i, j) queries as follows, similar to [35].We construct the suffix tree T(T ) and preprocess it so that each node stores the lexicographic range of suffixes of which its path-label is a prefix.We also construct a 2D orthogonal range counting data structure over an We thus maintain D using the solution of Proposition 6.7.
In order to answer a ReportDistinct(i, j) query for D, we first answer an analogous query for D .For each reported pattern min D[ p] ∈ D , we iterate over patterns P ∈ D[ p] in the order of increasing lengths, as long as P occurs T [i . .j]; we check whether this is the case using Theorem 6.6.Note that the relevant elements of each collection D[ p] are retrieved efficiently in the right order since D[ p] is implemented as a min heap.
Report(i, j) We first perform a ReportDistinct(i, j) query and then find all occurrences of each returned pattern in T [i . .j] in time Õ(1 + |output|) by Theorem 6.6.
Exists(i, j) We observe that the answer for D is the same as for D .For the latter dictionary, we use the static version of Count(i, j), presented in Sect.5, and the counting version of internal pattern matching (Theorem 6.6) for the removed/inserted patterns, incrementing/decrementing the counter appropriately.We rebuild the data structure every m updates.
Count(i, j) We first build the data structure of Sect. 5 for Count(i, j) queries for dictionary D 0 .For the subsequent m updates, we use this data structure to answer Count(i, j) queries, treating individually the inserted/removed patterns using Theorem 6.6.These queries are thus answered in Õ(m) time.
After every m updates, we update our data structure as follows to reflect the current dictionary.(We focus on D 0 and D m for notational simplicity.)We update the counts of occurrences for all nodes of PT by computing the counts for the set of inserted and the set of removed patterns in Õ(n) total time and updating the previously stored counts accordingly.
As for Breakpoints-Anchor IDM, we also have to do something smarter than simply recompute the whole data structure from scratch, as we do not want to spend Ω(d) time.At preprocessing, we set our grid G to be of size K × K for K = O(n 2 ) and identify x-coordinate i with the ith smallest element of the set W = {U x : U a substring of T and x ∈ {$, #}}.(Similarly for y-coordinates and T R .) In order to compute the rectangles in G and transform queries to points as in the original solution, we need to be able to efficiently transform strings in W to their ranks.
Let us show how to preprocess the suffix tree T(T ) in O(n) time so that the rank of a given T [a . .b]$ or T [a . .b]# in W can be computed in Õ(1) time.Let us assume that T(T ) has been built for T , without the $ appended to it.We make a DFS traversal of T(T ), maintaining a global counter cr, which is initialized to zero at the root.The DFS visits the children of a node in a left-to-right order.When traversing an edge, we increment cr by the length of the path-label of this edge.When an explicit node v with path-label S is visited for the first time, we set the rank of S$ equal to cr, and when v is visited for the last time, we set the rank of S# to cr + 1.Let q be the locus of T [a . .b] in T(T ), which can be computed in O(log log n) time using a weighted ancestor query.If q is an explicit node, the ranks of T [a . .b]$ and T [a . .b]# are already stored at q. Otherwise, these ranks can be inferred from the ranks of S$ and S#, where S is the path-label of the nearest explicit descendant v of q by, respectively, subtracting and adding the distance between v and q.
Thus, rectangles and points from the Proof of Lemma 5.4 are maintained in the K × K grid G.After m updates, we remove (resp.add) the Õ(m) rectangles corresponding to patterns in D 0 \D m (resp.D m \D 0 ).Finally, instead of using Theorem 5.1, we can maintain a data structure of size O(r ) for the counting version of range stabbing in a 2D grid of size K × K with r rectangles with O(log K log r /(log log r ) 2 ) time per update and query [25].Since we are not optimizing Õ(1) factors in the complexity, range trees (cf.[10]) can also be used for 2D range queries to avoid randomization.
To wrap up, updating the data structure every m updates to D requires Õ(1 + n/m) amortized time.We can deamortize the time complexities using the time-slicing technique.This concludes the proof of the following theorem.Theorem 6. 8 For every text T of length n and parameter m ∈ [1 . .n], a dynamic internal dictionary D can be maintained in Õ(n + |D|) space, supporting Õ(n + |D|)-time initialization, Õ(n/m)-time updates, and Õ(m + |output|)-time queries Exists(i, j), Report(i, j), ReportDistinct(i, j), and Count(i, j).

Fig. 1
Fig. 1 Occurrences of patterns from the dictionary D in the text T

Fig. 2
Fig. 2 Example of a D-modified suffix tree for dictionary D = {aa, aaaa, abba, c} and text T = adaaaabaabbaac from Example 1.1.Top: the suffix tree T(T ) with the nodes corresponding to elements of D annotated in red; bottom: the D-modified suffix tree of T 2 and 3.5, and the data structure answering range minimum queries in O(1) time can be built in time O(n)[9,24].(b)We first identify all positions a ∈ [i . .j] that are starting positions of occurrences of some pattern P ∈ D in T [i . .j] using RMQs over array Shortest(D) as follows.The first RMQ is over the range [i . .j] and identifies any such position a (if any such position exists).The range is then split into two parts, [i . .a − 1] and [a + 1 . .j].We recursively use RMQs to identify the remaining positions in each

Fig. 3
Fig. 3 An illustration of array I k and the colored range reporting query that we perform.The k-dictionary consists of the blue, green and red patterns, whereas C k consists of the blue and red patterns.Note that patterns that start in [i . .t) cannot end in a position to the right of j First, we perform a colored range reporting query on the range [i . .t) of array I k and obtain a set C k of distinct patterns, employing Theorem 4.1.(An illustration is provided in Fig. 3.) We observe the following.

Fact 4 . 4 (Lemma 4 . 5
Sparsity of occurrences) The occurrences of a pattern P of an aperiodic k-dictionary D aper k in T start more than 2 k 3 positions apart.Proof If two occurrences of P started d ≤ 2 k 3 positions apart, then d would be a period of P, contradicting P ∈ D aper k .ReportDistinct(t, j) queries for the aperiodic k-dictionary D aper k and j − t < 2 k+1 can be answered in O(1 + |output|) time with a data structure of size O(n + |D aper k |) that can be constructed in O(n + |D aper k |) time.Proof Since the fragment T [t . .j] is of length at most 2 k+1 , it may contain at most three occurrences of each pattern in D aper k by Fact 4.4.We can thus simply use a Report(t, j) query for dictionary D aper k and then remove duplicates.The complexities follow from Theorem 3.6(b).

Fig. 4
Fig.4 All the runs in a sample string T , annotated with their lengths and periods

Lemma 4 . 9
If a pattern P occurs in a text Q and satisfies |P| ≥ per(Q), then P has exactly one occurrence in the first per(Q) positions of Q.
per k -modified suffix tree of T and the array k := Shortest(D per k ) in O(n + |D per k |) time (using Lemmas 3.2 and 3.5, respectively).We then preprocess the array k for RMQ queries.Answering a query Let us consider a query for U = T [t . .j].We first compute all runs in R k (U ) using the following lemma.Lemma 4.10 Let U be a fragment of T of length at most 2 k+1 .Then R k (U ) can be retrieved in O(1) time after an O(n)-time preprocessing.

Fig. 5
Fig. 5 Distinct patterns in D per k that are generated by R ∈ R 4 (U ) and occur in R ∩ U are reported by their occurrences in the first per(R) positions of R ∩U .Dashed occurrences are not reported as the corresponding patterns occur earlier

Lemma 4 .
11 ReportDistinct(t, j) queries for the periodic k-dictionary D per k and j − t < 2 k+1 can be answered in O(1 + |output|) time with a data structure of size O(n + |D per k |) that can be constructed in O(n + |D per k |) time.Overall, by summing over all k ∈ [0 . .log n ], we obtain a data structure of total size O(n log n + d) that can be constructed in time O(n log n + d) and answers aper k , D per k for k ∈ [0 . .log n ]: (a) the D -modified suffix tree with a data structure for level ancestor queries [8], which use O(n + d ) space, (b) arrays Shortest(D ) and Longest(D ), which use O(n) space, and data structures for answering RMQs on these arrays, (c) the data structure for colored range reporting on an auxiliary array I k , which uses O(n) space.The remaining data structures-for answering 2-period queries (in the construction of D aper k and D per k ) and periodic extension queries (Lemma 4.10), and the sets of runs R k -take O(n) space in total.Let us now show how to modify the components (a)-(c) to reduce the space usage to O(n + d) without losing the desired functionality.Part (a) We store the D-modified suffix tree and, for all k ∈ [0 . .log n ] and i ∈ {aper, per}, we mark all nodes from each D i k with a different color.For each dictionary D ∈ {D k , D aper k , D per k }, we further store, using O(|D |) space, the D -modified suffix tree with its leaves trimmed, augmented with the level ancestor data structure.
We build an O(N )-bits RMQ data structure over J .The query procedure however, needs access to A, i.e. the colors.Let us recall that our array of colors I k is defined by assigning to all the patterns ofD k equal to T [a . .L k [a]] for some a, where L k := Longest(D k ), colors id in [1 . .n],and setting I k [a] = id(P), where P = T [a . .L k [a]].The array L k can be retrieved from part (b).Let us recall that each entry L k [a] can be retrieved in O(1) time along with the corresponding internal node of the D k -modified suffix tree.All the internal nodes are stored explicitly, so their colors can simply be kept with them.With Lemmas 4.3, 4.5, 4.11 and the above modifications to the space usage, we arrive at the main result of this section.

Theorem 4 .
12 ReportDistinct(i, j) queries can be answered in time O(log n + |output|) with a data structure of size O(n + d) that can be constructed in time O(n log n + d).

Observation 5 . 3
The number of fragments T [r + 1 . .t] = P satisfying i ≤ r + 1 ≤ t ≤ j and β − r ∈ B is equal to the number of rectangles stabbed by the point of the grid defined by u and v .The observation holds because this point is inside rectangle R b for b ∈ B if andonly if P[ b . .m] is a prefix of T [ β . .j] and P[1 . .b ] is a suffix of T [i . .β ].This concludes the proof of the following result.Lemma 5.4 Breakpoints-Anchor IPM queries can be answered in O( log n log log n ) time with a data structure of size O(n + |B|) that can be constructed in time O(n + |B| √ log |B|).Next, we define the analogous Breakpoints-Anchor IDM problem, a variant of the Internal Dictionary Matching (IDM) problem.Breakpoints-Anchor IDM Input: A length-n text T , a dictionary D of patterns, and for each pattern P ∈ D, a set B P of inter-positions (breakpoints) of P. Query: Count β (i, j): for a given inter-position (anchor) β in T , the number of fragments T [r + 1 . .r + |P|] contained in T [i . .j] that satisfy T [r + 1 . .β ] = P[1 . .b ] and T [ β . .r + |P|] = P[ b . .|P|] for some P ∈ D and b ∈ B P .(Equivalently, T [r + 1 . .r + |P|] = P and β − r ∈ B P .)

Lemma 5 . 5
Breakpoints-Anchor IDM queries can be answered in O( log n log log n ) time with a data structure of size O(n + P∈D |B P |).The data structure can be constructed in time O(n + √ log n P∈D |B P |).
[a . .b], which is a sequence of symbols labelling a certain chain of nodes whose values constitute T [a . .b]. Theorem 5.6 ([41]) If two fragments match, then their popped sequences are equal.Moreover, each popped sequence consists of O(log n) runs (maximal powers of a single symbol) and can be constructed in O(log n) time.The nodes corresponding to symbols in a run share a single parent.Furthermore, the popped sequence consists of a single symbol only for fragments of length 1.Let us now provide some more intuition.For any occurrence T [a . .b] of a string S in T , we have a chain of nodes whose labels can be represented compactly, whose values constitute T [a . .b], and whose labels are independent of what precedes or succeeds T [a . .b].This yields a uniform handle for all such occurrences.
[1 . .k].The space consumption is O(n + d log n) since |B P | = O(log n) for each P ∈ D.Efficient preprocessingThe RLSLP and the parse tree are built in O(n) time, and the sets B P are determined in O(d log n) time using Theorem 5.6.The data structure of Lemma 5.5 is then constructed in O(n + d log 3/2 n) time.

Fig. 7
Fig. 7 Node v has two children, u and z.We denote val(u) by a grey rectangle, and val(z) by a green rectangle.T [a . .c] is the longest prefix of S = T [a . .b] that is contained in val(u).Note that the labels of v 3 and v 4 must be distinct as these nodes do not share a single parent-one is a proper descendant of u, while the other is a proper descendant of z.Hence, |T [a . .c]| ∈ L(S) using a weighted ancestor query; we can have precomputed the leftmost occurrence of the path-label of each explicit node of T(T ) in a DFS traversal.We can initialize D in O(n + |D 0 |) time by answering all weighted ancestor queries as a batch[32,  Section 7].The dictionary D = {min D[ p] : 1 ≤ p ≤ n}, where min D[ p] is the shortest pattern whose length is stored in D[ p], is of size O(n).Let P be a pattern inserted to or deleted from D, and let p be the position of the leftmost occurrence of P in T .For such an update in D, we update D[ p] and (possibly) update D as follows.-CaseI: P is inserted to D. If P is shorter than min D[ p] or D[ p] is an empty collection, then P is inserted to D and min D[ p] (if it exists) is deleted from D .Finally, P is also inserted to the collection D[ p].-Case II: P is deleted from D. If P = min D[ p], then P is deleted from D , P is deleted from D[ p], and the new min D[ p] (if any) is inserted to D .Otherwise (if P = min D[ p]), P is only deleted from D[ p].Observe that if P ∈ D[ p] occurs in T [i . .j], then min D[ p] also occurs in T [i . .j].

Table 1
Our results 2)+d)-size data structure that answers CountDistinct(i, j) queries in Õ(m) time.-Wecan construct in Õ(n + d) time a data structure that gives 2-approximate answers to CountDistinct(i, j) queries in Õ(1) time.
In an earlier version of this work, we presented an Õ(n + d)-size data structure that could answer CountDistinct(i, j) queries O(log n)-approximately in Õ(1) time, by appropriately adapting our data structure for Count(i, j) queries.As this data structure is now obsolete, we have not included its description in this version.

that can be
Arrays Shortest, Longest for T and D = {aa, aaaa, abba, c} from Fig.2.
constructed in O(n + d) time.(b) Report(i, j) queries can be answered in O(1+|output|) time with a data structure of size O(n + d) that can be constructed in O(n + d) time.Proof (a) It can be readily verified that the answer to query Exists(i, j) is true if and only if the minimum element in the subarray Shortest(D)[i . .j] is at most j.Thus, in order to answer Exists(i, j) queries, it suffices to construct the array Shortest(D) and a data structure that answers range minimum queries (RMQ) on Shortest(D).The array Shortest(D) can be constructed in O(n + d) time using Lemmas 3.