MELArt: A Multimodal Entity Linking Dataset for Art

Given the need to ensure precise matching and integration of data across data sources, to efficiently merge Artpedia with the two other online databases, Wikidata and Wikimedia Commons, a structured approach is necessary. Our process can be systematically broken down to:

1. 1.

   Locating the artwork’s Entity-ID by Title: Identify and locate the specific entity in Wikidata using its Artpedia title (which is equivalent to its Wikipedia article and unique inside Artpedia). This step is implemented by querying Wikidata’s links to Wikipedia articles using the following SPARQL query²²2All the Wikidata extraction is performed on a dump downloaded on 2024-09-13 and generated on 2024-09-06T23:44:15Z installed on QLever [4] as SPARQL engine.:

   ⬇

   PREFIX schema: <http://schema.org/>

   SELECT ?wikipedia_title ?wikidata_id WHERE {{

   VALUES ?wikipedia_title {"@title"@en}.

   ?wikipedia_id schema:name ?wikipedia_title.

   ?wikipedia_id schema:about ?wikidata_id.

   ?wikipedia_id schema:isPartOf <https://en.wikipedia.org/>.

   }}

   where $\mathrm{@}title$ is the title from Artpedia. In this process we found that for a subset of 20 paintings, the match was not possible. The two main reasons for a missing match were: i) deleted Wikipedia articles like “Charles II in armor” and ii) Wikipedia articles that became disambiguation pages like “The Three Musicians (painting)”. For those cases we manually defined the matching between Artpedia’s title and Wikidata.

2. 2.

   Extract from Wikidata, using SPARQL, the following predicates for each artwork:

   • $\mathrm{■}$

     P18: image. We extract the URL of the first image for each painting that is part of Wikimedia Commons.

   • $\mathrm{■}$

     P180: depicts. This corresponds to each “entity visually depicted in an image, literarily described in a work, or otherwise incorporated into an audiovisual or other medium”.

In this step, we take advantage of the rich set of alternative labels defined in Wikidata entities to consider the multiple surface forms that might be present in Artpedia artwork description sentences to refer to depicted entities. To do so, we extract Wikidata’s alternative labels for the depicted entities. For each of the depicted entities, we list all the available alternative labels. For example, the entity Virgin Mary (Q345) can be referred to by multiple surface forms, which include among others: Our Lady, The Virgin Mary, Madonna, and Holy Virgin.

At the end of this process, the total number of labels for depicted named entities becomes 12,966, averaging 2.9 labels per depicted entity.

In this stage, we create a rich set of mention-entity pairs by finding the depicted entity labels inside Artepdia’s sentences. This pair construction is a critical step, as it forms the backbone of the MELArt dataset, linking the entities mentioned in the artwork descriptions to their corresponding entities within a structured and accessible knowledge base. This structure is critical for facilitating research and applications in art and cultural heritage data analysis. The steps we use are the following:

1. 1.

   Select named entities. The depicted entities identified in the previous step are carefully filtered to preserve named entities and remove general entities. Given the set of all the entities depicted in an artwork, we decide to only keep named entities as iconographic statements. In our process, this is important because automatically matching non-named entities might create false positives since their surface forms are common words. Similar to [9], to determine if an entity is a named entity (i.e., an iconographic statement), we extract the English labels, and check if any of them contain an uppercase character. In this way we avoid entities like horse or hand and keep entities like Christ or Paris. An initial evaluation of using Wikidata entity types (i.e., using instance_of predicates) as a filter for named entities, with a manually defined set of classes like human and city, showed smaller coverage of entities as compared to the capitalization heuristic. An inspection of the missed entities showed that the selection of the set of classes needed to cover all the depicted named entities in the dataset is a manually intensive and error-prone task. We thus refrain from manually defining and maintaining such an entity type set and instead follow the more automated process that looks at letter capitalization.

2. 2.

   Match the labels to the artwork sentences. For each of the paintings, we match any of the labels of the depicted named entities to the visual and contextual sentences from Artpedia, using the Spacy’s Phrase Matcher³³3https://spacy.io/api/phrasematcher. The result is a set of spans in the sentences corresponding to the depicted entities. Given that the list of possible entities is limited by the depicted named entities, this approach focuses on high matching precision. The matching is case-sensitive, to avoid false positives, especially with names like Our Lady which are composed of common words.

3. 3.

   Filter nested mentions. In the case of overlapping matches, where one match is completely contained in another match, we select the longer and more specific entity mention. For instance, if the entity Madonna and Child is recognized as an entity, but in the same span Madonna is also recognized, then the latter will not be considered.

In this stage, given the recognized and linked mentions, we build an extensive set of Wikidata entities that can be potentially linked to the depicted entities. We base our expansion on the assumption that entities sharing labels are challenging to disambiguate, with a limited context.

Table 1 illustrates various entity candidates for the name “Gustave Courbet”. It exemplifies the multifaceted challenges inherent in entity disambiguation tasks within art datasets. The table shows the wide variety of meanings that can be connected to a single name, highlighting how one term can have many different interpretations or associations. The labels in the table range from the person “Gustave Courbet” himself to various exhibitions themed around him, and even streets named after him. This example shows how widely a single name can be interpreted, pointing to different entities or concepts each time it is mentioned. The descriptions confer granularity, distinguishing the historical individual from specific art expositions and location entities. Crucially, the “Instance of” category elucidates the ontological nature of each referent, spanning classifications like “Temporary Exhibition”, “Print”, and “Scholarly Article.” This typological diversity underscores the need for an advanced semantic analysis approach, incorporating label and feature extraction (from both text and image) to ensure precision in the mapping of nominal references to their corresponding entities within a knowledge base such as Wikidata. The steps we use are the following:

1. 1.

   Wikidata labels full-text index. We first index all the English entity labels from Wikidata into an Apache Solr⁴⁴4https://solr.apache.org/ core. This system is configured to index the text of the labels using an English analyzer composed of tokenization, stop word removal, possessive filter and stemmer. In addition to the labels, we include the number of Wikipedia articles for each entity to represent the popularity of the entity.

2. 2.

   Expand the candidates list using the mentions’ text. Given the set of matched mentions in the sentences, we expand the set of candidate matching entities for those mentions using the indexed Wikidata labels. For each mention (e.g., “Mary”) we perform two queries to the Solr index. First, we find the 25 most similar labels to the mention, based on the BM25 algorithm (k1=1.2,b=0.75). This returns candidates with very similar labels like “Mary, Mary, Mary” (Q113570633). Additionally, we perform the same search but sorting the results according to the number of Wikipedia links in descending order and choosing the first 25 results. This query finds entities like “Marie Curie” (Q7186) with only a partial label match, but with a high popularity. We then include these two sets of candidates and the ground truth entity to the universe of MELArt candidates. This final set contains 53,901 entities.

3. 3.

   Extract details from the candidates. For each of the candidate entities, we use SPARQL to extract the following predicates.

   • $\mathrm{■}$

     P18: image. For each candidate entity, we extract all Wikimedia Commons images, except those that correspond to the Artpedia artwork images. We filter those because for some candidate entities, the related images already correspond to the paintings in which they have been depicted. If we were to keep them, that would make the entity linking task trivial in those cases. For instance, the painting Ophelia(Q21192340) by John William Waterhouse depicts the character Ophelia(Q1800888) from Hamlet, but the image associated with the Wikidata entity of the character is Waterhouse’s painting.

   • $\mathrm{■}$

     P31: instance of. We extract the classes the entity belongs to, and their main labels.

   • $\mathrm{■}$

     We extract Wikidata’s English description of the entity.

4. 4.

   Download images. Together with the painting image links, and all the candidate images, the image files are downloaded from Wikimedia Commons or Wikipedia in a few cases. Wikimedia Commons links are given priority in case there is are multiple images for a painting.

To evaluate the quality of MELArt, we follow a multistep process with two human annotators that manually complete the entity recognition and linking tasks on a sample of the painting description sentences. The goal of this evaluation is to answer the following question: are the automatically annotated mentions precise and exhaustive? Ideally, all the annotations should correspond to correct entity mentions, and they should be linked to the right Wikidata entities.

We limit the scope of this evaluation to annotations of linked Wikidata entities, thus avoiding the identification of depicted entities not linked in Wikidata, even though there are hints in the text indicating their presence. For instance, for the painting Rucellai Madonna (Q948745) and the visual sentence “The painting depicts the Virgin and Child enthroned, surrounded by angels on a gold background.” an annotator could identify Madonna and Child (Q9309699) as a good fit for the “Virgin and Child” span, however, according to Wikidata’s triples, this artwork depicts Virgin Mary (Q345) and Christ Child (Q942467) as independent entities. In such cases, we assume Wikipedia’s links as the ground truth and refrain from adding extra links.

In total, MELArt covers 1,616 artworks split in training (1,188), validation (328) and test (100) sets.

The dataset resulting from the process described above was constructed from 7,170 visual and 14,526 contextual sentences from Artpedia, contrasted with the matching of 2,118 visual and 2,716 contextual sentences in MELArt. The reduction in number of sentences is due to many sentences not directly mentioning the depicted entities (e.g., “The couple had been married for only a brief period”). The numbers also show a slightly stronger representation of visual sentences (44%) in MELArt as compared to Artpedia (33%). This is expected given that depicted entities are more likely to be mentioned in visual than contextual sentences.

The final MELArt statistics after introducing the manual annotations are:

• $\mathrm{■}$

  Number of mentions: 6,585

• $\mathrm{■}$

  The training, validation, and test splits correspond to: 4,632, 1,308, and 645 mentions respectively.

• $\mathrm{■}$

  2,916 mentions are associated to visual sentences and 3,669 to contextual sentences.

In Figure 4, we can see that although there are especially long contextual sentences, the trend in terms of number of mentions and number of words per sentence remains similar among visual and contextual sentences.

Considering the split of the dataset, we can see in Figure 5 that in terms of number of sentences for each artwork, the automatically annotated training and validation sets show similar trends. The manually annotated test set, however, has slightly more annotated sentences per artwork. This can be interpreted as a result of the lower recall of the automatic annotations, resulting in less mentions per artwork in the automatic annotations as compared to the manually annotated test set. When training a model with MELArt, this does not impose a problem in terms of the format of the training and validation data points, but raises an interesting challenge about linking potentially unseen surface forms that can be found in the test set.

In terms of the artworks in the dataset, we can leverage the information in Wikidata to understand potential biases in the dataset sources and generated annotations. Figure 6 shows a histogram representing the periods in which the included artworks were created⁵⁵5Making use of Wikidata’s inception date (P571).. We can see three strongly represented periods: one covering the 15th and 16th centuries, one for the 17th century, and lastly the 19th century.

By observing the movements and genres in Figure 7, it is clear that the Renaissance is the most common movement in the dataset, being also tightly related with the religious art genre. The genres show a stronger skewness towards religious art and portrait, which might be a result of focusing on named entities in MELArt. Analyzing the category distribution of Artpedia paintings before and after matching with the Wikidata “Depicts” predicate, reveals interesting distribution differences. In the Artpedia dataset, “religious art” (660 paintings), “portrait” (585), and “genre art” (259) are the most popular genres, while in the MELArt dataset, the dominance of “religious art” (571) and “portrait” (295) persists. Notably, “genre art” sees a significant drop from 259 to 121 in the matched dataset. Categories like “mythological painting” and “history painting” show a smaller decrease in representation. The matched dataset also exhibits a drastic reduction in categories like “landscape art” and “animal art”, highlighting a potential under-representation of these genres in MELArt due to the lack of named entities depicted in them. This comparison reveals a skew in MELArt’s representation of art, with a strong focus on religious and portrait art, while other genres like “landscape art” being less prominently featured due to the scarcity of named entities in them. This reflects possible biases and limited art genre diversity in both Artpedia and MELArt.

In Figure 8 we analyze the origin of the artworks according to the artists, where it is clear that the dataset is strongly skewed towards European art. Another clear bias in the dataset is related to the gender of the artists: out of 468, 413 are male, 27 are female and 1 is a non-binary/queer artist, according to gender information in Wikidata.

We also illustrate statistics about the depicted subjects in the annotations in Figures 9 and 10. We observe that although entities from different classes⁶⁶6We restrict our analysis to Wikidata classes directly related to the mentions. are mentioned in the sentences it is more common to find humans and its subclasses than others, e.g., geographical features.

In terms of concrete entities, aligned with the genres seen in Figure 7, religious entities are very prominent, with Mary (Q345) being the most frequently mentioned entity in the dataset. We also see that the dataset is skewed towards a group of entities, having half of the mentions linked to the most frequent five percent of the 1,306 mentioned entities, and eighty percent of the mentions being covered by one third of all the entities.

All the artworks have an associated image in the dataset. Roughly half of the full candidate set contain at least one image. In the subset of mentioned candidates more than 90% contain at least one image.

In this section, we present the evaluation of tree entity linking methods using the MELArt dataset. To this end, each method ranks the candidates according to its specific scoring function, and we leverage on common ranking evaluation metrics to compare them. The resulting scores are sorted in descending order to compute Hits at k ($H\mathrm{@}k$), Mean Reciprocal Rank ($MRR$), and Mean Rank ($MR$). These metrics are computed as follows:

The metric $H\mathrm{@}k$ reflects the presence of the correct entity within the top $k$ entities ranked by score. MRR denotes the average of the inverse of the rank of the correct entity, and MR represents the average rank of the correct entity relative to all entities. Thus, higher values of $H\mathrm{@}k$ and MRR correspond to better performance, whereas for MR, a lower value denotes superior performance. The measured ranks are computed per mention, meaning that if a sentence has two entities mentioned, each one is ranked independently.

In order to showcase the use of our dataset to evaluate multimodal entity linking method effectiveness on art data, we tested one entity and relation linking tool for Wikidata and trained and tested two state-of-the-art (SOTA) entity-linking methods using our MELArt dataset.