Unified Multimedia Segmentation – A Comprehensive Model for URI-based Media Segment Representation

The Synchronized Multimedia Integration Language (SMIL) is an XML-based language to create interactive multimedia presentations [34]. Users can include multimedia, like audio, video, and text, and define timings, transitions, and animations. For that, SMIL supports both temporal and spatial fragments. For the former, the user can specify the beginning and end of the desired snippet of a longer video or audio object. For the latter, SMIL allows cropping rectangles from images or videos. Additionally, transitions of the rectangle can be defined, which enables the creation of a primitive spatio-temporal fragment.

The Scalable Vector Graphics (SVG) format, also based on XML, is used to describe two-dimensional vector and raster images [40]. Besides rectangles, arbitrary paths can be specified, allowing for any fragment shapes. Additionally, with svgView, fragments can be established directly in the URI of an SVG object, although only rectangular regions are supported. Temporal segments are not naturally supported, but SVG allows adding HTML or SMIL as foreign objects, which themselves could contain content with a temporal component.

The MPEG-7 standard is created by the Moving Picture Experts Group (MPEG) for describing multimedia content of many different types [37]. Unlike previous MPEG standards, which focus on storing content, MPEG-7 provides means to store information about the content. MPEG-7 defines four building blocks [3, 65, 41]:

• $\mathrm{■}$

  Descriptors describe syntax and semantics of a feature that characterizes the content.

• $\mathrm{■}$

  Description schemes describe the structure of relationships between components (descriptors or description schemes).

• $\mathrm{■}$

  The description definition language allows creating descriptors and description schemes as well as extension and modification of existing description schemes.

• $\mathrm{■}$

  Syntax tools are used for binarization, synchronization, transport, and storage of descriptions.

The Segment description schema can be used to annotate temporal, spatial, and spatio-temporal fragments using XML-based metadata [65]. Temporal fragments are specified using starting time and duration. Spatial regions are created from polygon coordinates, allowing for arbitrary shapes. It is also possible to construct a segment from multiple components. On top of that, time frames and regions can be combined into spatio-temporal fragments.

[7] proposed the Semantic Web as an extension to the World Wide Web. Its goal is to share data, resulting in a Web of Data on the web of documents of the traditional web. The Semantic Web architecture makes use of various standards. XML, for example, can be used to create structure in webpages by annotating sections with tags, but it says nothing about the meaning of those structures [10]. For that, the Resource Description Framework (RDF) is used [13]. In the form of triples (i.e., (subject, predicate, object) tuples), assertions between things and properties are made, and a knowledge graph is built. The elements of the triples are either literals or URIs, with the latter pointing to further information about a resource. This Linked Data principle enables information linking and ensures unique definitions of concepts [4]. RDF alone does not prevent different systems from using different terms for the same concept. For computers to understand and reuse information across systems, shared vocabularies and ontologies, which define common concepts and relationships, are needed. A vocabulary is usually a simple description scheme specified in RDF Schema (RDFS) [29]. More complex semantics are usually modeled in the Web Ontology Language (OWL) [17]. Lastly, the SPARQL query language [68] for RDF provides an SQL-like syntax to extract data.

At the same time, multimedia content on the Web has increased, and newly created media formats, as described above, allow metadata annotations. These are, however, not easily understandable by computers, as the different standards lack formal semantics [70]. To address this issue, Semantic Web technologies are required, and a number of ontologies for single media and multimedia as well as several different ontologies to annotate various types of multimedia documents have been proposed.

For image description, for instance, [49] proposes a vocabulary to describe regions of various shapes, such as polygons, rectangles, or circles, and link them to images together with additional information. Numerous ontologies based on MPEG-7 have been proposed to describe the semantics of general multimedia. The first attempt is made by [35], who models parts of MPEG-7 using the ABC core ontology by [45] as a foundation. [76] builds on this and extends it to an ontology called DS-MIRF. [36] takes a slightly different approach in their audio-visual core ontology and focuses more on adding genre or themes to segments. [9] proposes a framework that uses the core ontology DOLCE [25] with two multimedia ontologies, namely Multimedia Structure Ontology and Visual Descriptor Ontology. [26] is the first to produce a full MPEG-7 ontology, called Rhizomik, by automatically mapping XML Schema to OWL. [2] proposes COMM, a core ontology for multimedia annotation. Unlike previous ontologies, it does not directly translate MPEG-7, but the authors re-engineer MPEG-7 to capture the intended semantics of the standard fully. This ontology covers the most used parts of the specification.

Beyond MPEG-7, the Ontology for Media Resources 1.0 defines a core vocabulary to describe media resources and a mapping to a set of media formats that support metadata [67]. Its goal is to unify metadata properties across commonly used multimedia formats. The Multimedia Metadata Ontology (M3O) by [64] also looks beyond MPEG-7 and provides a generic modeling framework that can be integrated with media formats like SMIL or SVG.

In the Hypertext Markup Language (HTML), URI fragments are used to reference anchor elements and allow jumping to specific content on the same or another page. Anchors are created using either the name or id attribute. Such a fragment URI, for example, looks like this: http://example.com#section-1. To resolve it, the fragment name is compared to all element IDs of the document tree, and the first matching element is used [32]. To solve the problem of changing IDs and broken references, text fragments can be used instead, which directly reference textual content [12]. Being a relatively recent specification, it is only implemented in Chromium-based browsers at the time of writing.

In XML, the XPointer Framework defines ways of addressing specific components through URI fragments [28]. Besides the shorthand form for referencing specific element names, two schemes allow advanced references. The element() scheme can be used to traverse the XML tree and target specific children [27] (e.g., example.xml#element(/1/4) ). The xpointer() scheme extends the functionality even more and allows operations on ranges and strings, allowing highly specific addressing of segments [18] (e.g., example.xml#xpointer(string-range(//title, ’text’, 4, 5))).

[78] proposes the usage of URI fragment identifiers for plain text files. Its proposal, later partly captured in [79], includes the char and line schemes, used either as single positions or ranges. For example, the segment of lines 11 to 20 is identified as: http://example.com/text.txt#line=10,20.

Additionally, the length and md5 schemes are provided for integrity checks. In a similar fashion, [31] defines URI fragment schemes to select specific rows, columns, and cells in comma-separated values (CSV) documents. For example, a segment of cells that starts at the second row and second column and ends at the eighth row and sixth column is targeted like this:
http://example.com/data.csv#cell=2,2-8,6

MPEG-21 is another standard developed by the MPEG, focusing on creating, delivering, and consuming multimedia [38]. The key components of the MPEG-21 framework include digital items created by and exchanged between users. Digital items are virtual containers for resources, which include both multimedia objects as well as metadata [11]. One such example is a music album, which, besides the actual audio tracks, may include lyrics, copyright information, cover art, and metadata about the songs or the artist.

Most relevant for this paper, part 17 of the specification defines URI-based fragment identifiers [39]. With four different schemes, which can also be combined, specific parts of resources can be addressed:

• $\mathrm{■}$

  ffp() identifies items and tracks by their name or ID.

• $\mathrm{■}$

  offset() is used to indicate the start and, optionally, the length of a byte segment of the resource.

• $\mathrm{■}$

  mp() is applicable to audio-visual resources and can be used to specify temporal, spatial, or spatio-temporal segments in numerous ways.

• $\mathrm{■}$

  mask() applies a binary mask to a video resource.

The following example applies both a temporal and a spatial segmentation to a video:
video.mp4\#mp(~time(’ntp’,’0:00:05’,’0:00:15’)*mp(~region(rect(100,100,200,200)))

While these schemes provide powerful means to address segments without indirections, [52] deems them over-designed and ambiguous due to their complexity. For instance, both ffp() and mp() can be used to segment a track. Additionally, only MPEG formats are supported.

Temporal URIs have been proposed as part of the Continuous Media Markup Language (CMML) and the file format Annodex, which together enable hyperlinking and annotating continuous media, like audio or video [57, 58]. A user can express one or multiple temporal segments by specifying start and end times: video.mp4?t=15/25. The specification allows both URI fragments and URI queries to be used depending on the needs.

All file standards described above, although providing means to specify temporal and spatial segments, have certain problems associated, like indirections, complex expressions, or too narrow scopes. Therefore, [73] concludes that a standardized way to address segments is needed, which should follow the cool URIs practices published by [16]. This allows media content to become a first-class citizen on the Web. Based on this proposition, the Media Fragments URI 1.0 W3C specification was created [75, 52]. URI fragments are chosen over URI queries, as fragments keep the relation to the original. That way, a media fragment is of the same file type as the parent resource. This setup ensures that media fragments follow the Linked Data principles [30]. The specification includes four fragment dimensions, which, due to being logically independent, can be combined freely:

• $\mathrm{■}$

  Temporal fragments are set with start and end time, where different time formats are supported: video.mp4#t=5,15

• $\mathrm{■}$

  Spatial fragments are limited to rectangles, specified as pixel coordinates or in percentages. The rectangle is distinctly defined by xywh, i.e., the position of the top left corner pixel and width and height of the bounding box: image.jpg#xywh=200,200,100,50

• $\mathrm{■}$

  One or multiple tracks can be segmented from media objects with multiple tracks (e.g., video, audio, subtitle track) by specifying track names: video.ogg#track=audio

• $\mathrm{■}$

  The id dimension is used to select a named fragment, such as a chapter from an audiobook: audio.mp3#id=chapter-1

In major browsers, however, only temporal fragments are natively supported.²²2https://bugs.chromium.org/p/chromium/issues/detail?id=94368
https://bugzilla.mozilla.org/show_bug.cgi?id=648595
https://trac.webkit.org/changeset/104197/webkit Using the other dimensions requires custom software. As far as we were able to ascertain, there are also no other applications, such as dedicated media players, that natively support media fragments. In addition to that, existing Web infrastructure can only use byte ranges to refer to parts of a resource, like a temporal or named fragment. To resolve and retrieve media fragments over HTTP, different strategies are proposed in the specification and in literature [19, 22, 21, 48, 80].

• $\mathrm{■}$

  No mapping: As defined in [5], URI fragment identifiers are only interpreted by the user agent after the full resource is downloaded from the server. This approach, which can be found in browser implementations, only requires the user agent to be able to interpret and render media fragments. On the downside, this wastes bandwidth, as potentially large parts of the resource are discarded upon retrieval.

• $\mathrm{■}$

  User-side mapping: The user agent maps fragments to byte ranges. This is possible when the user agent has already downloaded parts of the resources, like headers or an index, and knows about the mapping. Then, a partial content request for this specific range can be sent to the server.

• $\mathrm{■}$

  Server-side mapping: The user agent requests the fragment in a special header, and the server performs the mapping. While special extensions must be installed on the server, this solution is the most efficient for bandwidth and latency. [19] implements a prototype in NinSuna, a platform for multimedia content selection and adaptation [20].

• $\mathrm{■}$

  Third-party mapping: A Media Fragments Translation Service (MFTS) [48] performs the mapping to byte ranges, either as a service for the user agent or as a proxy between a user agent and server. This approach requires minimal changes to the existing infrastructure.

While proper implementation of the full specification hence requires different client and server considerations, there exist various media management and retrieval systems that rely on media fragments, like the works of [42], [56], or [66]. The last, for example, implements a prototype system that can first segment videos of parliament speakers using video and audio segmentation techniques and second present the segments together with additional annotations on parliamentary websites. Beyond that, [47] uses media fragments for adaptive streaming over HTTP.

The media fragments URI specification exhibits a number of limitations, such as no spatio-temporal fragmentation or spatial fragments being limited to rectangles. To address these limitations, extensions to the specification were proposed by [43]. Those extensions provide more options for spatial fragments, like circles, ellipses, or polygons. They also introduce static and animated shape transformations, which can be applied to spatial fragments. These additions enable more precise fragment specifications, as well as spatio-temporal fragments. The authors also provide a CSS extension to style fragments directly. On GitHub, more similar extensions can be found.³³3https://github.com/tomayac/dynamic-media-fragments
https://github.com/oaubert/mediafragment-prototype

While media fragments can be used in Semantic Web technologies, multimedia-specific queries are not supported. Hence, [44] presents SPARQL-MM, an extension that provides spatio-temporal filter and aggregation functions to SPARQL. This allows, for example, querying a video snippet that contains two specific persons, as long as the persons are identified by a fragment URI. [55] follows a similar idea and provides support for temporal SPARQL queries with media fragments. They introduce query logic that can compare temporal fragments to, for instance, query events that happened before a certain event.

The International Image Interoperability Framework (IIIF), as described by [71], is both a set of standards for image delivery and a community of cultural heritage institutions, like museums, libraries, and archives. Its goal is to allow the sharing of digital images using Linked Open Data principles. It provides application programming interfaces (APIs) for interoperable access to images and metadata stored in different repositories. At its core, IIIF has two APIs: The Presentation API provides metadata in the form of a JSON manifest. It describes a IIIF item (e.g., a book, newspaper, or artwork) as a collection of references to resources (e.g., scans of individual pages) and defines its structure and layout. Additionally, it can contain annotations, where media segments are typically referenced following the Media Fragments URI specification. Various IIIF clients exist that can load and display these manifests. Images are queried through the Image API with standard HTTP requests. A specific image and desired characteristics are specified in a URI following the format: <image API>/<identifier>/<region>/<size>/<rotation>/<quality>.<format>. The identifier references the image, region defines a rectangular segment, either as pixel coordinates, percentages, or “full”. Size and rotation are used to scale and rotate the region, respectively, while quality allows turning the image grayscale or black and white. Lastly, format specifies the image file format. The client application uses the region specifier to only fetch the image section currently in view. An advantage of this is that the browser can work with standard HTTP requests and does not need to distinguish between statically served and dynamically generated images. Thus, unlike when relying on the Media Fragments URI specification, the browser requires no additional logic to retrieve partial images. This fact makes it convenient to use IIIF image URIs in the general Semantic Web context independently of the presentationAPI [51].

With IIIF, the image server does all the work. It prepares the requested image segment from the original according to the request URI and sends it back over HTTP. The requested images are generated “on the fly” from high-quality images and transformed into the requested format. There exist several image server implementations online⁴⁴4https://iiif.io/get-started/image-servers/ and in literature [61].

It has been recognized that IIIF is limited to images and should be extended to support other media formats, such as audio and video. To that end, [15] coins the acronym IxIF to represent all non-image resources and provides an interim implementation; an audio and video (A/V) working group has been set up subsequently. The initial timetable was to extend the presentation API and then move on to content APIs. For the latter, [59] looks into an IIIF-based video framework and conducts some experiments on encoding and decoding speeds of videos, which is crucial for real-time delivery. Later, the working group decided to only focus on the presentation API, as it was deemed much more important. Nonetheless, there exists one account of an IIIF-like API for A/V content [50]. This API defines the request syntax <A/V API>/<identifier>/<region>/<size>/<rotation>/<temporal_region>/<quality>.<format>,where the added temporal_region parameter determines the time interval and controls playback speed.

The EPUB Canonical Fragment Identifier (EPUB CFI) specification defines ways to access specific content inside an electronic publication (EPUB) document [72]. Such a reference code can, for example, look like this: book.epub#epubcfi(/6/4[chap01ref]!/4[body01]/10[para05]/3:10). Slashes are used to navigate the document tree, where even and odd indices describe child elements and chunks of character data, respectively (/6 therefore denotes the third child element, while /3 refers to the second chunk of data). Exclamation marks perform an indirection step by accessing another document referenced in the current element. Square brackets assert the ID of the current element, and colons are used for character offsets. While the standard is generally used to extract XML or text elements, it also features identifiers for temporal, spatial, and spatio-temporal offsets. A fragment identifier in this format tends to become complicated quickly and only applies to the EPUB format, thus further limiting its applicability as a fragment representation.

The Open Data Protocol (OData)⁵⁵5https://www.odata.org/ is a REST-based protocol for defining CRUD operations. It enables the encoding of filtering operations and function calls into HTTP URLs. An example for retrieving a filtered list of documents authored in or after 2024 that contain the term “Painting” in their title could look like this:
/OData/Documents?\$filter=AuthorDate geq 2024-01-01 and contains(Name, "Painting").
While OData is primarily used for structured data encoded in JSON, it has also found uses in hypermedia applications.

Most segmentation tasks revolve around wanting to retain a subset of the data points of the initial media object. The filter operation is thus generally defined as

where the filter function

is an indicator function that decides for each data point in the space whether it should be kept or not. It can use any properties of the data point itself or its description to make the decision. Examples of such filter segmentations are the selection of a rectangular cutout of an image, a scene from a movie, an excerpt of an audio recording, or one page from a PDF document.

The filter segmentation is not limited to a single filter function but can consist of an arbitrary number of sub-filters $f_1,\mathrm{\dots },f_k$, which are combined using Boolean algebra operations. For only conjunctions, the filter operation is defined as

For arbitrary Boolean operations, data points are part of the segment if the Boolean combination of indicator functions yields $1$. This allows the mixture of any filter criteria, for instance, only keeping image pixels of certain colors, which are located in a certain area of the image and do not have large illumination changes compared to neighboring pixels. From Equation 8, it follows that

using the definition of set intersection. Equally, by the definition of set union, it follows that

This shows that the conjunction and disjunction of sub-filters are equivalent to the intersection and union of segments, respectively, where each segment is created using one particular sub-filter. The same relationship can also be shown for arbitrary combinations of operators.

As the segmentation result is another media object, multiple filter segmentations can be applied consecutively, i.e.,

Thereby, the multimedia object is increasingly filtered by every segmentation. This corresponds to applying all filter operations in the same segmentation operation, which can be written as a conjunction of filters. As derived above, this is equivalent to the intersection of the segments that result from applying the filter segmentations to the original media object. It follows that nested filter segmentations are commutative:

As an illustration, consider a temporal video segment, which is afterward cropped to a square. Its result is equivalent to first cropping the whole video and then cutting it to the desired length. The segmentations can also operate on the same dimensions, such as in an image with overlapping segment areas, where the order of segmentation does not matter for the outcome.

Alternatively, a user might want to keep all data points but segment the description of each point. This reduction operation

applies a reduction function

to the description of each data point. Like the filter function, this dimensionality reduction function is not allowed to modify the individual data points but can only select which dimensions to keep or discard. A possible application could be extracting one color channel of an image.

The reduction function can be seen as a special filter function, which operates on only the codomain of the media object. However, it does not filter elements of the set but the description dimensions of each element. In other words, the scope of each data point description is changed. Hence, set theory operations are not applicable in a mathematical sense. However, one can interpret the description of a data point as an $m$-dimensional set instead of an $m$-tuple of values. Thereby, the reduction function can be described as a filter function on each set of values, enabling the combination by set operations. Then, the reduction operation is commutative. See Section 4.2 for a practical example.

Filter and reduction operations can also be combined but require caution, as commutativity is not always given. In some cases, the filter operation might use properties affected by a reduction operation, for example, a filter operation based on color and a reduction operation that drops specific color channels. In other cases, segmentations are commutative, like selecting one color channel and cropping the image. A solution to this uncertainty is adapting the filter operation to only use data point information (i.e., $P$) without changing its semantics. For instance, the filtering based on color can also be represented as filtering of certain point coordinates that happen to have said color. With this trick, filter and reduction functions operate on disjoint pieces of information, which ensures commutativity.

Besides operations that directly affect ordered pairs of the set, utility operations need to be considered. One issue is the fact that a certain media object can have different representations, where each one is defined in a different space. To ensure that any segmentation can be applied, the media object might need to be transformed in preparation. Such a transformation function

transforms a media object $M$ from one space to a media object $M^{\prime }$ in another space, such that $M$ is semantically equivalent to $M^{\prime }$. The two media objects can have different sets of dimensions. An example of that is the Fourier transformation of an audio signal from time and amplitude dimensions to frequency and magnitude dimensions. Any such transformation must be reversible, such that the inverse transformation can be applied to the segmented media object to restore the format of the original media object. Note, however, that such transformations can be lossy.

Postprocessing operations can be applied to the result of a segmentation operation. Strictly speaking, these are not part of the segmentation definition, as they modify the media object instead of just selecting a fraction of it. However, for real-world applications, postprocessing can be vital for usability, as it allows speeding up a song, rotating an image, performing anti-aliasing to smoothen edges, or removing a constant dimension. The last allows turning a single frame of a video, which is so far just a very short video, into an image by removing the temporal dimension. As these operations do not fulfill the segmentation definition, they are not discussed further in this context.

Segments can be described in terms of their boundary, as segment data points are usually coherent. Due to the limited amount of information needed, this approach is well suitable to be conveyed in a URI. This is supported by the observation that all URI-based segment annotation approaches presented in Section 2 describe segments in terms of boundaries. Depending on the dimensionality of the boundary, different considerations are necessary.

An alternative to the straightforward boundary definition is the usage of algebraic expressions, which describe hypersurfaces and can cover a variety of smooth shapes. As these expressions require little description space, they are a good candidate for short URIs. In Figure 13, where the equation is $y>-0.019x^2+9.6x-900$, all points above the curve are segmented as they fulfill the stated inequality. As a last step, the output can be cropped to the segment bounds, which is the whole image in this case.

If describing segment boundaries becomes too complex, an alternative, which allows data point-accurate segment description, is a binary mask. Such a mask can be described in a URI as an enumeration of binary values. This scheme is usable for any media object type, but an enumeration order needs to be established for media of more than one dimension. For images, this can, for instance, be line-wise from top to bottom, as shown in the example in Figure 14. When the mask is two-dimensional, it can alternatively be represented as a serialized binary image, which can additionally be base64-encoded to shorten the length of it drastically. In that case, no ambiguities must be dealt with, provided the mask matches the media object. This approach, also presented in Figure 14, corresponds to the Data-URL scheme specified in [53]. In the example’s case of only $10\times 10$ pixels, listing binary values is more efficient, but with larger masks, the encoded PNG becomes superior. In the worst case, when the mask is completely randomized, the base64 encoding is one sixth the size of the binary enumeration. Otherwise, the encoding can be multiple orders of magnitude smaller with more coherent masks.

Clear enumeration orders of data points are provided by space-filling curves, making its application rather straightforward. Due to its locality preservation properties, the Hilbert curve is chosen for this purpose. While such a curve is defined to visit every data point, one can instead play with the curve’s order. This determines the maximum index and thus controls the resolution of the segmentation. For example, a first-order Hilbert curve for two dimensions splits each dimension in half, yielding four sub-squares, each corresponding to one curve index. Therefore, the URI serialization of a segmentation needs to carry information about the order of the curve and a series of index ranges. An example of a third-order Hilbert curve applied to an image is given in Figure 15. It shows a roughly similar excerpt as the previous image segmentations, but precision could be improved by increasing the order and, hence, the number of index ranges. In the same way, Hilbert curves can also be used to segment media of higher dimensions, such as videos. In both cases, the media object can be cropped to the segment bounds, and data points not in the segment need to be neutralized.