Uncertainty Management in the Construction of Knowledge Graphs: A Survey

Formally, KGs are directed and labeled multigraphs $(ℰ,ℛ,𝒯)$, where $ℰ$ is the set of entities, $ℛ$ is the set of relations, and $𝒯$ is the set of triples $⟨\text{subject, predicate, object}⟩$, which are the atomic elements of KGs, also called facts. The subject and object are represented by nodes, the predicate indicates the nature of the relationship holding between the subject and object represented by an edge in the KG [69, 122]. For instance, such a triple could be $⟨\text{Suwon-si, location, Korea}⟩$ as illustrated in Figure 3. The classes and relationships of entities are defined by a schema or otherwise named an ontology, which itself can be represented as a graph embedded in the KG [42, 36]. To build a schema or ontology, different vocabularies or languages are available, such as RDF Schema⁴⁴4https://www.w3.org/TR/rdf12-schema/ (RDFS) or OWL (Ontology Web Language) and OWL 2 [52]. RDFS is designed to describe RDF vocabularies specific to a domain by defining classes, subclass relationships, properties, domain, range and sub-property relationships. OWL is an ontology language and an extension of RDFS defined by Description Logics (DLs), which are a family of formal knowledge representation that enable reasoning about ontologies. Specifically, OWL DL is based on the description logic $𝒮\mathscr{H}𝒪ℐ𝒩$ [52] and OWL 2 DL is based on $𝒮ℛ𝒪ℐ𝒬$ [70]. OWL and OWL 2 allow the creation of ontologies with more complex constraints than RDFS (e.g., cardinality restrictions). They include several sub-languages, each offering different levels of expressiveness depending on the constructors applied (e.g., disjunction, inverse roles, role transitivity, etc.) in the description logic that defines the language. In a KG with an ontology defined using a language based on DLs, two boxes coexist, namely the Terminology Box (TBox) and the Assertion Box (ABox). The TBox defines classes and properties and the ABox contains instances of classes defined in the TBox. For example, in Figure 3, “Company” and “Country” are concepts defined by the ontology, “Galaxy S23”, “Samsung”, and “Korea” are the instances of these concepts while “2023”, “800€” are literals i.e., attributes that characterize an entity. This semi-structured representation of data, defined by its ontology, provides a clear and flexible semantic representation whose classes and relations can be easily added and connect a large number of domains [36].

For the last few years, some KG construction projects that link general knowledge about the world have appeared. The best known is probably Wikidata⁵⁵5https://www.wikidata.org/wiki/Wikidata:Main_Page, a large, free, and collaborative KG supported by the Wikimedia Foundation. It is a multilingual general-purpose KG that contains more than 100 million elements [155]. The structure of Wikidata is based on property-value pairs, where each property and entity is an element. A typical entity also contains labels, aliases, descriptions, and references to Wikipedia articles. To maintain the quality of data, some constraints such as property or unique value constraints alert the user in case of suspicious input (e.g., constraint violations). Wikidata also keeps the sources and references of provenance of entities to ensure their traceability, which is one of the requirements for KG quality [162].

NELL is an intelligent computer agent that ran continuously between January 2010 and September 2018 according to the official NELL website⁶⁶6http://rtw.ml.cmu.edu/rtw/ and that every day extracted knowledge from texts, tables, and lists on the web to feed a Knowledge Base (KB) [19]. To maintain the consistency of the KB, the knowledge integrator of NELL exploits relationships between predicates by respecting mutual exclusion (e.g., an instance of a Human class cannot be an instance of a Car class, since Human and Car are mutually exclusive) and type checking information. In addition, NELL components provide a probability for each candidate and a summary of the source supporting it.

YAGO is an ontology built on statements of Wikipedia that combines high coverage with high quality [145]. The data model of YAGO is based on entities and binary relations extracted from WordNet and Wikipedia. A manual evaluation is performed to verify the quality of data. To do this, facts are randomly selected with their respective Wikipedia pages that are used as Ground Truth (GT).

DBpedia is a multilingual KB built by extracting structured information from Wikipedia (e.g., infoboxes) and making this information available on the Web [7]. Since DBpedia is populated from Wikipedia pages in different languages, the data retrieved can sometimes be conflicting. To manage these conflicts, DBpedia has a module called Sieve which performs quality assessments by computing some metrics such as the “recency” or the “reputation” of data before applying a fusion step based on these dimensions [111, 16].

Freebase is a graph created in 2007 which provides general human knowledge and which aims to be a public directory of world knowledge. A component included in Freebase called Mass Typer, allows users to complete and semi-automatically reconcile data with data already present in Freebase by performing three actions: merge, skip, or add the data. Then acquired by Google and used to support systems such as Google Search, Google Maps, and Google KG, nowadays, Freebase is closed, and its knowledge has been transferred to Wikidata [11].

ConceptNet is the KG version of the Open Mind Common Sense project that contains information about words from several languages and their roles in natural language. It was built by collecting knowledge from multiple data sources namely Open Mind Common Sense, Wiktionary, games with a purpose for harvesting common knowledge, Open Multilingual WordNet, JMDict, OpenCyc, and DBPedia. Each node corresponds to a word or a sentence, and the relations between nodes are associated with numerical values that intend to represent the level of uncertainty about the relation [142].

EKGs are major assets for companies since they can support various downstream applications including knowledge/vocabulary sharing and reuse, data integration, information system unification, search, or question answering [51, 122, 139]. This led companies such as Google, Microsoft, Amazon, Facebook, Orange and IBM, to build their own KGs [122, 77].

For instance, Microsoft built Bing KG to answer any kind of question through Bing search engine. With a size of about two billion entities and 55 billion facts according to [122], it contains general information about the world such as people, or places and allows users to take actions such as watching a video or buying a song. Alternatively, KGs can also increase understanding of user behavior.

It is the case of the Facebook KG that establishes links between the users as well as interests of users, e.g., movies or music tastes. The Facebook KG is the largest social graph with about 50 million entities and 500 million statements in 2019. To handle conflicting information, the Facebook KG removes information if the associated confidence is low, otherwise, conflicting information is integrated with its provenance and the estimated confidence of the information.

Yahoo KG [117] provides various services such as a search engine, a discovery system to relate entities, or for entity recognition in queries and text. To build their KG, they leverage Wikipedia and Freebase as the backbone of the KG and use various complementary data sources to maximize the relevance, comprehensiveness, correctness, freshness, and consistency of knowledge. They mainly validate the data w.r.t. the ontology and through a user interface that enables entities to be corrected and updated.

Also, Orange bootstraps its KG from a set of terms of interest from a enterprise repository [77]. These terms of interest are aligned with their equivalent Wikidata entities. Then, an expansion is performed by retrieving their N-hops neighborhood to identify additional entities of interest. To ensure the quality of this initial KG, pruning methods based on Euclidean distance in the embedding space, degrees of Wikidata entities, or a method based on analogical inference are used [77, 76].

In [122], the authors mention future challenges including disambiguation, knowledge extraction from unstructured and heterogeneous sources, and knowledge evolution management in the process of KG construction. We discuss some of these challenges in the next section.

Methods for populating a KG depend on the knowledge domain and the desired graph coverage. For example, one method could rely on the knowledge of domain experts and populate the graph manually (e.g., by crowdsourcing, such as Wikidata [155] or Freebase [11]). However, this is a time-consuming process, especially if the graph is intended to be large and may suffer from quality issues [15, 140]. Furthermore, such open KGs have a large community that enterprises or specific KGs may not have. Therefore, large KGs such as Google, Amazon and Bing rely on automatic construction methods [122]. In the following sections, we present the different tasks involved in knowledge extraction from various data sources.

Assessing the quality of the constructed KG is important since it is practically impossible to obtain a perfect KG, especially when it is very large and populated by automatic approaches from multiple data sources, or by manual approaches where human contributors are not necessarily familiar with KGs and have different levels of expertise. Furthermore, the world is uncertain and knowledge is constantly evolving. To evaluate a KG, we can rely on five quality dimensions [160, 170, 69, 67]: completeness, accuracy, timeliness, availability, and redundancy.
Completeness refers to the coverage of knowledge within the specific domain the KG is intended to represent. The evaluation of this dimension depends on the assumption made about the KG [50]: the Open World Assumption (OWA), the Closed World Assumption (CWA), and the Local Closed World Assumption (LCWA). Under the OWA, if a triple is not present in the KG, it is not necessarily incorrect but rather unknown. Conversely, under the CWA if a triple is not present in the KG, then it is considered incorrect. Finally, under the LCWA if the KG knows at least one object or value $v$ for a predicate $p$ associated with the subject $s$, it is assumed to know all the values of the pair $(s,p)$. For example, this dimension can be measured as the number of instances represented in the KG relative to the total expected number of instances. Some studies have focused on estimating the expected total number. More details on these measures can be found in [160].
Accuracy corresponds to the correctness of the facts in the KG. In [162], Weikum et al. define some metrics to assess the quality of a KB such as precision that captures the accuracy (these terms are sometimes used to describe the same thing), and recall that captures the completeness, in the following way:

where $S$ is a set of statements from the KB to be evaluated, and $GT$ is the ground truth set for the domain of interest. To deal with uncertain statements that are associated with a confidence score, a threshold is chosen, for which all statements with a score above this threshold are kept. They also provide an evaluation method that involves uniformly taking a sample of statements and representative of the KB and evaluating it, for example manually, where several annotators may be involved and a consensus or large majority must be found for each annotation.
Timeliness represents how up-to-date the KG is. The KG can contain temporal facts or facts that evolve and are valid only over a fixed period of time.
Availability measures the access to KG data, involving its querying and representation. For example, this dimension can be measured by the response time to queries or by the level of accessibility to KG data (e.g., RDF, Turtle, JSON-LD, or SPARQL query service).
Redundancy assesses whether different statements express the same fact, which may require an entity resolution task where duplicates are aligned and then the triples associated with these duplicates are fused.

Another aspect of data quality is the preservation and representation of its provenance and certainty in the form of metadata, which can be used for data selection issues in terms of both source and quality. The metadata can also support knowledge fusion approaches by serving as prior knowledge, as we describe in Section 8. Other metrics are proposed in the survey [160] for each quality dimension.

Uncertainty is everywhere in knowledge and can take the form of invalidity, vagueness, fuzziness, timeliness, ambiguity, and incompleteness according to [34]. We adopt this definition of uncertainty in this survey. We distinguish two types of uncertainty: epistemic, i.e., knowledge about a piece of information is incomplete or unknown; and ontic, i.e., uncertainty is inherent in the information [156]. The possible causes of uncertainty are [1, 156]: (i) a lack of knowledge; (ii) a semantic mismatch or a lack of semantic precision, and (iii) a lack of machine precision.

When a KG is constructed from multiple heterogeneous data sources, uncertainty can lead to the emergence of knowledge deltas, characterized by differences in the information or facts they contain. These knowledge deltas may occur between two data sources on the same subject, e.g., differences in specificity and contradictions. For example, a very specific data source $A$ and a generic data source $B$ may provide information on the same topic but with different terms, increasing the risk of knowledge delta. It is also possible for a data source to contradict itself, one possible way to detect these deltas is to compare the data source with itself by “reflecting on data patterns or extrapolation to complete missing information and/or detect wrong ones” according to [33]. On the other hand, duplicates can also occur when the two data sources provide exactly the same knowledge, which needs to be managed for reasons of scalability and KG quality.

We use some examples to illustrate the various forms that uncertainty can take. Suppose that $t$ is a statement. Among the possible knowledge deltas, we find six causes:

• $\mathrm{■}$

  Invalidity: $t$ is invalid. As illustrated in Figure 5 (a), the Wikipedia text provides invalid information: the date of renaming of the Paris region to “Île-de-France” is invalid in the Wikipedia page¹⁰¹⁰10https://en.wikipedia.org/w/index.php?title=Paris&oldid=1197869134;

• $\mathrm{■}$

  Vagueness: $t$ provides vague, imprecise information. As depicted in Figure 5 (a), the date mentioned on Wikipedia is more vague than the date provided by Wikidata¹¹¹¹11https://www.wikidata.org/w/index.php?title=Q90&oldid=2058313448 for the “located in the administrative territorial entity” property, which contains additional information such as the day, month, and year;

• $\mathrm{■}$

  Fuzziness: $t$ states a fuzzy truth, where the range of values is itself imprecise. If we focus only on the sentence within the black box in Figure 5 (b) of the Wikipedia article¹²¹²12https://en.wikipedia.org/wiki/5G, it indicates that the 5G network has a higher peak download speed, but without specified lower and upper bounds.

• $\mathrm{■}$

  Timeliness: a data source may provide the statement $t$ which is no longer valid at the current time, unlike another source, which may provide an updated version of $t$. As in Figure 5 (c), on the Wikipedia page¹³¹³13https://en.wikipedia.org/w/index.php?title=Twitter,_Inc.&oldid=1087087372 of May 10 2022, “Twitter” had not yet been renamed “X”. This information has now been changed, otherwise there would have been an update issue;

• $\mathrm{■}$

  Ambiguity: $t$ has multiple interpretations. As shown in Figure 5 (d), Mercury¹⁴¹⁴14https://en.wikipedia.org/wiki/Mercury can be a planet, an element, or a god in mythology;

• $\mathrm{■}$

  Incompleteness: $t$ gives incomplete information. As in Figure 5 (e), the tracklist of the album “Evolve” by the group Imagine Dragons on Wikidata¹⁵¹⁵15https://www.wikidata.org/w/index.php?title=Q29868187&oldid=2009666363 contains fewer songs than in Wikipedia¹⁶¹⁶16https://en.wikipedia.org/w/index.php?title=Evolve_(Imagine_Dragons_album)&oldid=1197244329.

The appearance of knowledge deltas can be involuntary or voluntary. An involuntary delta could be the result of uncertain knowledge about a domain (e.g., popular science article vs expert article), a typing error, or an outdated data source. A voluntary delta could simply stem from sabotage by a malicious person (for example, spreading fake news). Deltas are closely related to the quality dimensions of a KG, since they have a direct impact on them. For example, a delta due to the invalidity of an information from a data source directly affects the accuracy of a KG. We propose to classify these types of deltas leading to conflicts into two classes as depicted in Figure 6, namely Specificity that stands for a difference between two data sources in the specificity of knowledge and Contradictory that stands for an incompatibility of knowledge. We classify Fuzziness, Incompleteness, and Vagueness deltas in the specificity category. These deltas lead to different levels of specificity between the knowledge of two data sources. This knowledge is not necessarily wrong, but may be in conflict e.g., a city vs. a country to describe the location of an event. On the other hand, Invalidity, Ambiguity, and Timeliness deltas lead to contradictory knowledge, where some parts of the knowledge are necessarily wrong.

In [9], the authors distinguish two types of data conflicts from a data fusion perspective: contradictions and uncertainties. The authors define contradictions as follows: “a contradiction is a conflict between two or more different non-null values that are all used to describe the same property of an object” and uncertainties as follows: “an uncertainty is a conflict between a non-null value and one or more null values that are all used to describe the same property of an object”. We adopt the same definition of contradictions, but adopt a different definition of uncertainty. We define the second type of conflict as a difference in the specificity of knowledge, as illustrated in Figure 6. In this survey, “uncertainty” is a more general term whose sources lie in knowledge deltas and the inaccuracy of each step in the knowledge integration pipeline, including knowledge acquisition.

In [5, 177], the authors assume that uncertainty is a common feature of the knowledge we handle daily. In this sense, exploiting uncertain data sources by ignoring uncertainty to enrich a KG would impact downstream applications of the graph. The life cycle for exploiting uncertain data sources requires the quantification, and the integration of uncertainty in the KG. In such a view, the uncertainty should be considered everywhere in the data integration pipeline, including its representation in the KG. In Section 5.3, we present our ideal data integration pipeline that addresses the aforementioned requirements.

Integrating data from multiple sources can introduce inconsistencies and uncertainty into a database. Bleiholder and Naumann [9] describe the data integration process in three steps: 1) schema mapping, 2) duplicate detection, and 3) data merging. The first step establishes a common schema among data sources, the second step aligns duplicates and detects inconsistent representations for the same entity, and the final step combines and resolves the various inconsistencies (e.g., contradictions) to produce a unified representation. In [39], the authors mention that uncertainty can arise for several reasons, such as the approximation of semantic mappings between data sources and the mediated schema (i.e., the integration schema grouping all sources), the extraction techniques used to extract data from unstructured sources, and also mention uncertainty at the application level when querying with the transformation of keywords into a set of candidate structured queries. In [39, 136], the authors address the schema mapping step, considering uncertainty through probabilistic schema mappings and how to answer queries on the mediated schema. Different strategies can be used to deal with inconsistencies. One approach involves blaming the most recent assertion that caused the inconsistency in the KB [107]. In [28], Amo et al. consider two methods. The first is to allow inconsistencies to remain in the KB and reason about them using paraconsistent logic [8], while the second approach seeks to get rid of inconsistencies to obtain a coherent KB. Belief revision addresses the latter approach by updating the knowledge in a KB when a contradiction is encountered. The update is done by revising the KB, where some beliefs must be retracted [56]. The formalism used to represent beliefs and the nature of the relationship between explicit and implicit beliefs play an important role in the belief revision process [56]. In [107], Martins and Shapiro propose a tailored logic for belief revision systems. Their system tracks the support for each proposition in the KB and applies inference rules of the logic to compute dependencies between propositions. These dependencies help identifying sources of inconsistency and guide the revision process.

All ways of enriching a KG (e.g., crowdsourcing, extraction from texts or tables, etc.) are error-prone methods, since humans cannot be experts in every domain, involving mistakes and extraction algorithms rarely achieve perfect precision. Errors can occur at various stages in the data integration process that encompasses extraction, alignment, or fusion. Probably one of the most natural ways of capturing and quantifying uncertainty caused by knowledge deltas or the reliability of knowledge integration components is to use confidence scores. As mentioned in Section 4, several extraction approaches provide confidence scores about the triples they extract. For example, each triple outputted by ReVerb [46] is associated with a confidence score obtained from a logistic regression. Another work [96] focuses on estimating a confidence score for the slot filling task, which consists of filling predefined attributes for entities in a KB population case. This confidence score is intended to support the aggregation of values from different slot filling systems. The authors have shown that confidence estimation improves the performance of the task and that the correctness of the values and the estimated confidence are strongly correlated. In [163], the authors estimate confidence scores for an entity alignment task that represent the marginal probability that a set of mentions all refer to the same entity. Therefore, there is a need to consider these confidence scores and represent them as triple metadata along with their provenance information.

From this perspective, we propose an ideal pipeline of data integration from heterogeneous sources depicted in Figure 7. After extraction, knowledge integration often involves the following two modules [10]: Knowledge Alignment and Knowledge Fusion. In this pipeline, we propose a third module called Consistency Checking, which actually takes place after the data integration and identifies and repairs inconsistencies in the KG, improving future knowledge enrichment. The inputs of the pipeline are multiple heterogeneous sources whose final purpose is to feed the KG. From these data sources $s_j$, facts ${\mathrm{𝑓𝑎𝑐𝑡}}_{i,s_j}$ are extracted with different confidence scores such as a confidence score in the fact by the extraction algorithm ${\mathrm{𝑐𝑜𝑛𝑓}}_{\mathrm{𝑒𝑥𝑡𝑟𝑎𝑐𝑡}}({\mathrm{𝑓𝑎𝑐𝑡}}_{i,s_j})$, possibly a confidence score in the fact by the source ${\mathrm{𝑐𝑜𝑛𝑓}}_{\mathrm{𝑠𝑜𝑢𝑟𝑐𝑒}}({\mathrm{𝑓𝑎𝑐𝑡}}_{i,s_j})$, and a confidence score in the source $\mathrm{𝑐𝑜𝑛𝑓}(s_j)$. In addition to these multiple data sources, an expert can also populate the KG with a confidence score $\mathrm{𝑐𝑜𝑛𝑓}(s_e)$. Before providing these facts to the KG, several tasks are required due to potential knowledge deltas. The first task Knowledge Alignment is the identification of duplicates, differences in specificity, and contradictions between the extracted facts and the KG. The next step Knowledge Fusion defines a policy to resolve conflicting facts and keep the information as consistent, specific, and complete as possible. In addition, it estimates the quality of both data sources and extracted facts by assigning them confidence scores. Then, the knowledge in the KG is updated with its confidence scores and their provenance information since a user may want to query the KG about the confidence of triples w.r.t. quality dimensions. The last step checks the consistency within the KG. Since this step is performed after the enrichment of the KG, and since this survey focuses on uncertainty management in the construction of KGs, we do not provide further details on it in the following sections. The aim of this pipeline is to take into account all confidence scores in the knowledge alignment and fusion modules. This is not the case in existing work, where only the confidence scores in the data sources are leveraged by the fusion module. However, methods for KG completion purpose (i.e., predicting new relations using only the KG itself) taking into account the uncertainty in an embedding space have recently been investigated.

We describe the ideal data integration policy through an example. The integration pipeline takes as input a set of sources $𝒮$ containing a set of facts $ℱ$ and aims to construct a $𝒦𝒢$ as output. Each fact $f_{𝒮}\in ℱ$ is represented as a triple of the form $(e,r,v)$ where $e$ is an entity, $r$ is a relation, and $v$ is either a literal or another entity. The alignment of each fact with $𝒦𝒢$ consists in finding a correspondence between the pair $(e,r)$ from the fact and an existing $(entity,relation)$ pair in the current state of $𝒦𝒢$. If such a correspondence does not exist, the fact is added to $𝒦𝒢$ along with its associated confidence score and provenance information. Otherwise, if such a correspondence exists, i.e., $f_{𝒮}$ and $f_{𝒦𝒢}$ share the same $(entity,relation)$ pair, the pipeline applies the following integration policy:

1. (1)

   If $f_{𝒮}$ is more specific than $f_{𝒦𝒢}$, then $f_{𝒦𝒢}$ is replaced by $f_{𝒮}$ and the confidence score of the source $𝒮$ is increased. If such comparisons are not possible, the strategy is to keep both facts;

2. (2)

   Otherwise, if $f_{𝒮}$ is a duplicate of $f_{𝒦𝒢}$, only the provenance of $f_{𝒮}$ is added to the existing fact and the confidence scores of the sources providing it are increased;

3. (3)

   Otherwise, if $f_{𝒮}$ contradicts $f_{𝒦𝒢}$, the integration pipeline resolves the conflict by finding the most trustworthy value. The confidence of the source providing the least trustworthy fact is decreased, while the confidence score of the source providing the most trustworthy one is increased.

This integration policy aligns with the fusion approaches presented in Section 8.2. These approaches follow the intuition that the reliability of a source increases when it provides facts that are estimated to be correct, and a fact is estimated to be correct if it is supported by reliable sources. The implementation of this policy relies on the following key elements:

• $\mathrm{■}$

  Estimating specificity cannot be performed globally across all knowledge domains, as it is a context-sensitive and relation-dependent task. A KG does not have a total order, but contextualized partial orders can be deduced from the KG taxonomy when available with relations such as subclassOf, instanceOf, or partOf and can help determine whether one value is more specific than another [6, 81]. For example, (White house, location, United States) versus (White house, location, Washington): the most specific fact, namely $(\text{White house},\text{location},\text{Washington})$ should be kept by the integration policy. In addition, it may be interesting to leverage the generic information United States by deviating from it such as (White house, country, United States) to enrich the graph.

• $\mathrm{■}$

  Confidence scores from the upstream steps of knowledge fusion must be calibrated to represent them as probabilities on the same scale (e.g. via Platt scaling or isotonic regression) [35] to make them comparable and initialize the fusion models summarized in Section 8. Only the confidence scores estimated within the fusion model itself, namely those representing the reliability of the sources and the trustworthiness of facts are updated during the fusion process.

Algorithm 1 illustrates our vision of the policy for integrating conflicting data. It provides a straightforward guideline and corresponds closely to the framework of knowledge fusion models presented in Section 8.

As mentioned in the process above, we need to maintain a history of the provenance of the facts, as provenance information is essential for the quality of the KG, but could also be used for future conflict resolution or KG updating [73]. For this purpose, there exists a normative ontology called PROV-O [89] that includes provenance information. It can be used in RDF-based KGs. The three main classes of the PROV-O ontology are prov:Entity, prov:Activity, and prov:Agent (prov:Entity is something that can be changed by an activity, prov:Activity is something that acts upon or with entities, and prov:Agent can be a human who performs an activity).

In the following sections, we detail three steps of the pipeline namely: Knowledge Alignment, Knowledge Fusion, and Uncertainty Representation within the KG. We propose a section that describes uncertain KG embedding methods for KG completion and confidence prediction tasks (Section 6) before presenting the aforementioned steps. Knowledge alignment is discussed in Section 7. In Section 8, we summarize knowledge fusion methods. Finally, we explore the different mechanisms available for representing triple uncertainty in a KG in Section 9.

In this paper, we define uncertain KGs as follows. An uncertain knowledge graph (UKG) is represented as a set of weighted triples $𝒢=(s,p,o,s_t)$, where $(s,p,o)$ is a triple representing a fact and $s_t\in [0,1]$ is a confidence score for this fact to be true. The uncertainty associated with triples in the KG relies on the plausibility of the triples, but most KG embedding (KGE) methods do not consider this information in their modeling, making the assumption that all triples are deterministic. Such an assumption does not reflect the reality where many triples are uncertain due to the reasons described in Section 5. Table 1 summarizes the UKG embedding approaches with their associated tasks, scoring function, the year of publication, and the datasets on which the experiments were conducted. We can notice that uncertain graph embeddings have been studied only recently. Each of the following models has its own specific approach to incorporating a confidence score into their modeling. However, most models use a scoring function derived from existing deterministic KG embedding models, incorporating confidence scores, for instance, into the loss function used to train the embeddings.

UKGE [25] improves traditional KGE models by using the Probabilistic Soft Logic (PSL) framework to infer confidence scores for unseen relational triples. Thus, UKGE encodes the KG according to confidence scores for both observed and unseen triples. It maps the scoring function results to confidence scores using two different mapping functions: a logistic function and the bounded rectifier function. For the tasks of fact classification, global ranking, and confidence prediction tasks, UKGE outperforms the deterministic KG embedding models such as TransE, DistMult, ComplEx and the URGE model on CN15k, NL27k and PPI5k datasets.

SUKE [158] argues that UKGE does not fully exploit the structural information of facts. To improve this, SUKE has two components: an evaluator and a confidence generator. The evaluator assigns a structural score and an uncertainty score to each fact, which are jointly used to define the rationality of a fact. This rationality score is then used to generate a set of candidate triples, which are then fed into the confidence generator. The generator outputs confidence scores for these triples based on the uncertainty score provided by the evaluator. The plausibility of facts is computed with the DistMult [171] scoring function, then it applies a different mapping function with two parameters for the structural score and the uncertain score before merging them. The confidence generator only uses the uncertainty score to approximate the true confidence value of triples.

BEURRE [24] models entities as probabilistic boxes and relations between two entities as an affine transformation. The confidence score of the relation between two entities is represented as the volume of the intersection of their boxes. Constraints such as transitivity and composition are inserted into the modeling of embeddings to preserve these properties on relations in the embedding space. These constraints act as a loss regularization in the global loss function. Then, embeddings are trained by optimizing a loss function for a regression task and a regularization loss to apply transitivity and composition constraints.

GTransE [83] embeds uncertainty in a translational model by extending the well-known TransE [12]. Uncertainty, represented by a confidence score, is incorporated at the level of the loss function on a hyperparameter of the margin loss function when training the embeddings:

where $(h,r,t,s)$=(head, relation, tail, confidence score), M is a margin parameter and ${\left[x\right]}_{+}$ represents the positive part of $x$. The scoring function $f$ corresponds to the L1 or L2-norm. Thus, with this loss function if a triple $(h,r,t)$ has a high confidence score, it will tend to satisfy $h+r=t$, otherwise the entity $t$ will tend to diverge from $h+r$. Before GTransE, the same authors introduced CTransE [82] which is a related model that omits the hyperparameter $\alpha$ used as an exponent of the confidence score.

IIKE [47] models confidence within the embedding space using a probabilistic model. The authors propose an embedding model that takes uncertainty into account by minimizing a loss function to fit the output confidence of triples acquisition (e.g., NELL, or crowdsourcing) to the scoring function of the triples given by a probability function. The plausibility of a triple is modeled as a joint probability of the head entity, the relation and the tail $Pr(h,r,t)$ depending on $Pr(h|r,t)$, $Pr(r|h,t)$, and $Pr(t|h,r)$. For the loss function, the authors minimize the difference between the logarithm of the triple probabilities and the logarithm of the confidence scores from knowledge extraction. They then apply stochastic gradient descent to refine the embeddings iteratively.

PASSLEAF [26] decomposes the model into two components: a confidence score prediction framework that adapts scoring function from existing models, e.g., ComplEx [154] or RotatE [150], and a semi-supervised learning framework.

For the UKG completion task, each relation must have sufficient training examples to perform correctly. GMUC [178] tackles the few-shot UKG completion task for long-tail relations. GMUC learns a Gaussian similarity metric that enables the prediction of missing facts and their confidence scores from a limited number of training examples. The model encodes a support set, comprising a few facts along with their confidence scores, and a query into multidimensional Gaussian distributions. The query consists of (head, relation) pairs where the tail and confidence score must be predicted. A Gaussian matching function is then applied to generate a similarity distribution between the query and the support set. GMUC outperforms the UKGE model on link prediction and confidence prediction tasks on NL27K and three NL27K-derived datasets with added noise.

UOKGE [14] learns embeddings of uncertain ontology-aware KGs based on confidence scores. It encodes an instance $e_i\in E$ as a point represented by an n-dimensional vector, a class $c_i\in C$ as a sphere $s_i(c_i,{\rho }_i)$ where $c_i$ denotes the center of the sphere and $p_i$ its the radius, and a property $p_i\in P$ as a sphere $s_i((p_i^d,p_i^r),{\varrho }_i)$ where $(p_i^d,p_i^r)$ defines the center of the sphere, with $p_i^d$ representing the domain, $p_i^r$ representing the range, and ${\varrho }_i$ is the radius. It then introduces a mapping function to rescale values between 0 and 1 to represent uncertainty. Six distinct gap functions are defined to encode uncertainty for six types of relations: type, domain, range, subclass, sup-property, and remaining properties. The model then minimizes the mean squared error (MSE) between the confidence scores and the corresponding gap functions.

FocusE [124] improves KG embeddings with numerical values on edges by intervening between the scoring function of traditional models (e.g., TransE, ComplEx, or DistMult) and the loss function. They introduce numerical values on edges in a manner that maximizes the margin between the scores of true triples and their corruptions. Given the score function of an embedding model $f(t)$, where $t$ is a triple, they use a nonlinear Softplus function $\sigma$ such that the score provided by $f(t)$ is greater than or equal to zero:

The numerical value associated with an edge is then expressed through $\alpha$ as follows:

where $\beta$ is a hyperparameter controlling the importance of the topological structure of the graph and $w$ is the numerical value on the edge. The final function of FocusE is then: $h(t)=\alpha g(t)$.

ConfE [182] encodes the tuples $(e,\tau )$, where $e$ is an entity and $\tau$ is an entity type, by considering the uncertainty in each tuple. They treat entities and entity types as distinct elements in a KG and learn embeddings of entities and entity types in two separate spaces with an asymmetric matrix to model their interactions. The scoring function is defined as $G(e,\tau )=e^{\top }M\tau$, where $M$ is the asymmetric matrix. Uncertainty is incorporated into the loss function as follows:

where $\mathscr{H}$ is the set of entities and their types, and ${\mathscr{H}}^{\prime }$ is the set of corrupted tuples.

CKRL [168] introduces multiple levels of confidence, namely a local triple confidence, a global path confidence, a prior path confidence, and an adaptive path confidence. These confidence scores are integrated into the energy function as follows:

where $E(h,r,t)=\Vert h+r-t\Vert$ and $C(h,r,t)$ the triple confidence score aggregating all levels of confidence.

WaExt [86] embeds triples of a KG by incorporating the weight $w$ associated with an edge $(h,r,t)$ into the scoring function as follows:

and then minimizes a margin ranking loss function.

Wang et al. [159] model each entity and each relation as a multidimensional Gaussian distribution $𝒩(\mu ,\mathrm{\Sigma })$, where $\mu$ is a mean vector representing its position and $\mathrm{\Sigma }$ is a diagonal covariance matrix representing its uncertainty.

MUKGE [101] aims to improve the generation of unseen facts for KGE training. The authors argue that PSL cannot leverage global multi-path information, leading to information loss when estimating the confidence of unseen facts. Indeed, PSL only considers information from simple logical rules with a path length of two, as used in UKGE [25] (e.g., $(\mathrm{𝑐𝑜𝑙𝑙𝑒𝑔𝑒},\mathrm{𝑠𝑦𝑛𝑜𝑛𝑦𝑚},\mathrm{𝑢𝑛𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦})$ $\wedge$ $(\mathrm{𝑢𝑛𝑖𝑣𝑒𝑟𝑠𝑖𝑡𝑦},\mathrm{𝑠𝑦𝑛𝑜𝑛𝑦𝑚},\mathrm{𝑖𝑛𝑠𝑡𝑖𝑡𝑢𝑡𝑒})$ $\to$ $(\mathrm{𝑐𝑜𝑙𝑙𝑒𝑔𝑒},\mathrm{𝑠𝑦𝑛𝑜𝑛𝑦𝑚},\mathrm{𝑖𝑛𝑠𝑡𝑖𝑡𝑢𝑡𝑒})$), and does not consider other paths in the graph between the subject and object of the inferred relation. To address this issue, MUKGE introduces an algorithm called Uncertain ResourceRank, which infers confidence scores for unseen triples based on the relevance of the entity pairs (subject, object). The relevance of an entity pair is computed with respect to the directed paths between subject and object in the KG. MUKGE uses circular correlation as the scoring function, and applies either the sigmoid or the bounded rectifier function as the function to obtain the triple confidence. Then the authors design the loss function to align each positive triple to its corresponding confidence score. The authors evaluate their model on confidence prediction, relation fact ranking, and relation fact classification. For these three tasks, MUKGE outperforms the BEURRE and UKGE models, particularly focusing on asymmetric relations. However, for all relation types, the performance is competitive with other models, except for confidence prediction, where MUKGE is the best alternative.

In the literature, five datasets derived from uncertain KBs are commonly used in UKG completion or confidence prediction tasks, as presented in the previous section [124]. CN15K is a subset of ConceptNet (discussed in Section 3.2) where the numerical values represent the uncertainty of the triples [142]. The confidence scores for each triple are computed w.r.t. the number of sources and their reliability. NL27K is a subset of NELL dataset (presented in Section 4.1.2) where the confidence scores are computed and refined using an Expectation Maximization (EM) algorithm and a semi-supervised learning method. PPI5K is a KG that represents the protein-protein interactions, where the numerical values indicate the confidence of the relations [151]. O*NET20K is a dataset introduced by [124], containing descriptions of jobs and skills. The numerical values represent the strength of the relations. Some embedding models also generate their own synthetic noisy datasets with fictitious confidence scores following specific probability distributions.

The knowledge fusion step involves combining various pieces of information about the same entity or concept from multiple data sources into a consistent and a unified form, addressing the different deltas listed in Section 5.1 [67, 121]. The authors of [40] identify three broad goals to be achieved in this challenging task:

• $\mathrm{■}$

  Completeness: measures the expected amount of data (number of tuples and number of attributes) at the output of the fusion task;

• $\mathrm{■}$

  Conciseness: measures the uniqueness of object representations in the integrated data (number of unique objects and number of unique attributes of objects);

• $\mathrm{■}$

  Correctness: measures the correctness of data, i.e., its conformity to the real world.

Therefore, data fusion involves resolving conflicts in the data to maximize these three goals. Indeed, when integrating knowledge from multiple heterogeneous sources, the quality of the information varies, and we need to determine the trustworthy information by performing a Truth Inference (TI) task. According to Rekatsinas [132], different TI strategies can be adopted. There are simple strategies that estimate the true values of the entities compared to other values provided by the sources by applying a majority vote or an average on them. Additionally, there are strategies that use the trustworthiness of the sources to quantify the true values of the objects, and it is even possible to establish a precision metric for each class of object and for each source. The problem with simple strategies is that they do not take into account the varying quality of data sources [94], but they are often used to initialize the true values to start iterative TI methods.

For example, suppose that knowledge in the form of triples (subject, predicate, object) has previously been extracted about the cell phone “Galaxy S23” from several sources ${𝒮}_1$, ${𝒮}_2$, …, ${𝒮}_n$ resulting in the table at the bottom of Figure 8, where entities have already been aligned. We also represent the table as a graph for sources ${𝒮}_1$ and ${𝒮}_2$.

Several papers on data fusion use the term “data item”, i.e., (entity, attribute, value) instead of the term “triple” to refer to an element to be merged. However, in practice a data item is equivalent to a triple (subject, predicate, object). Each row of the table corresponds to an entity of a graph and its attributes, for example, the entity $e_1$ corresponds to the node “Galaxy S23” of the graph and the values associated with $e_1$ in the table are the objects of the triples. These objects are linked to the subject “Galaxy S23” by the predicates identified in the column headers depicted in Figure 8. The data extracted from both sources are almost the same except for the relations $software$ and $inception$, where the differences in specificity appear (e.g., the price of the cell phone). Source ${𝒮}_3$ states that the brand of the cell phone is “Apple”, contradicting “Samsung” provided by the first two data sources. Another example of a contradiction is the invalidity inception date of the phone provided by ${𝒮}_1$, which states it is 2021, whereas the correct date is 2023. We can also distinguish two levels of specificity. The first level concerns literals, such as numerical values or textual descriptions. In this case, specificity refers to the level of detail provided by the value. For example, the description of the Galaxy S23 provided by ${𝒮}_2$ contains more specific information than in the description provided by the first source, as illustrated in Figure 8. The second level concerns concepts. For instance, the concepts Korea and Asia may both be used to indicate the location of Suwon-si but they differ in specificity since Korea is more specific than Asia. Such differences can often be inferred from a taxonomy, as shown in Figure 10. Moreover, different levels of value representation can occur across sources, as illustrated in Figure 8. For instance, source ${𝒮}_1$ provides the triple $⟨\text{Galaxy S23, hasPrice, “800€”}⟩$, while source ${𝒮}_2$ provides $⟨\text{Galaxy S23, hasPrice, “expensive”}⟩$. The two values are not on the same representation scale: the first is a numerical value expressing the exact price of the cell phone, while the second is a qualitative assessment of the price. In all cases, the first and second sources provide complementary information for other attributes. Figure 9 shows the resulting graph after the reconciliation step that includes the knowledge alignment and the fusion step where the most complete representation of the Galaxy S23 entity is produced.

In the next section, we survey several methods that address knowledge fusion for truth discovery. We will use the terms “truth inferring”, “truth finding” or “truth discovery” interchangeably.

Fusion methods are listed in Table 3, which provides an overview and indicates whether the methods can handle certain characteristics of data or data sources such as numerical data, categorical data, data specificity, and dependencies between data sources.

SLiMFAST [132] leverages domain-specific knowledge features to improve the quality estimation of data sources. For example, if the data are extracted from scientific articles, the authors suggest using features such as the number of citations to the article or the year of publication, which can influence the quality of the source. Domain-specific features are incorporated into the parameters of the logistic function that estimates source quality. To merge the data, they apply statistical learning to estimate source quality and then apply probabilistic inference to predict the true values. To estimate the parameters, they use either the EM algorithm if the user does not provide labeled ground truth data or the Empirical Risk Minimization algorithm if the user does.

ACCU [37] incorporates the interdependence between data sources in the truth discovery process. The intuition is that a single source could provide the true value, while all the other sources could provide false values, knowing that some of them may copy from each other and therefore spread false values. Thus, if a data source provides a value different from all the others, it is not necessarily false. They define the dependency between two sources if part of their data comes directly or transitively from a common source, and it is computed using Bayesian models. Then, to discover the true value, they combine this dependency evaluation with the accuracy of the data sources, which is computed in relation to the confidences of the values and dependencies between sources.

POPACCU [41] is a refinement of the ACCU model. Unlike ACCU, it assumes that the data sources are independent and that only one value can be correct. In [41], the authors propose to select an optimal subset of data sources in order to maximize the quality of integrated data while minimizing data integration costs based on the Marginalism principle in economic theory. The model takes into account the distribution of incorrect values for an entity, based on observed values. The probability of a source providing an incorrect value depends on the popularity measure of the values. Unlike the basic model, the POPACCU variant is shown to be monotonic, i.e., adding a source never reduces the accuracy of the fusion.

CRH [94] (Conflict Resolution on Heterogeneous Data) estimates the reliability of a source by using all types of data simultaneously, instead of focusing on a single type. To do this, the authors use a loss function that measures the distance between the unknown truth and the values claimed by the sources for each type of data. To initialize the source reliability score, it first applies a simple conflict resolution method, such as majority or average voting, and models the estimation of source and truth weights as an optimization problem.

MDC [97] takes into account the semantic aspect of values. To illustrate the importance of semantics, the authors provide the following example: one data source provides the true value “common cold”, another source claims the value is “sinus infection”, while a third claims the value is “bone fracture”. Instead of examining all values at the same level, MDC calculates the semantic proximity among them. This semantic proximity calculation allows to evaluate how close a value is to the true value. The semantics of the values are captured through their vector representations, learned by following the idea that if two values share similar words, then their vectors should also be similar.

DOCS [183] is a system deployed on the Amazon Mechanical Turk, which takes into account the precision of the answers of each worker to assign tasks from specific domains to the most suitable worker. In terms of true value inference, the system uses the inherent relationships between the reliability of workers (viewed as data sources) and the true value. Thus, it considers two events: (1) let $v$ be the value of an entity provided by a source $s$. If the quality of the values provided by $s$ for entities in the same domain as $v$ is high, then $v$ is likely correct; (2) if a source $s$ often provides correct values for a domain $d$, then $s$ has a high reliability for domain $d$.

TruthFinder [176] claims that a fact is more likely to be true if it is provided by a reliable source, and that a source is reliable if it provides verified facts. Thus, an interdependence between facts and sources emerges. Consequently, TruthFinder uses three elements for its iterative trust discovery process: the trustworthiness of sources, the confidence of facts, and the influences between facts. The trustworthiness $t(w)$ of a source $w$ is computed by

and the confidence $s(\mathrm{t})$ of a fact $\mathrm{t}$ is computed by

The logarithm is then applied to simplify computation and obtain the final scores. Although simple, these definitions account for the influence of facts where values with varying levels of vagueness (e.g., “Joe Biden”, “J. Biden”, and “Biden”) can increase the confidence of a fact. They also take into account the dependence between data sources through a dampening factor acting on the computation of the confidence score.

While most existing methods consider more generic values of a correct one as false, TDH [81] leverages the hierarchical structure of knowledge (i.e., one aspect of specificity) to fuse data extracted from different sources. The idea behind this is that multiple values in the hierarchy of an entity could be correct even if the predicate is functional. For example, in Figure 10, “Korea”, and “Asia” are correct values for the location of “Suwon”, even though one of both values is more specific. Such modeling should not negatively affect the assessment of source reliability. Instead of classifying the value as $correct$ or $incorrect$, they introduce three classifications: exactly correct, hierarchically correct, and incorrect. Therefore, TDH models each source not only by its reliability but also by its individual tendency to generalize or be specific, which are jointly learned in the TI process.

In the same way, ASUMS [6] adapts existing truth discovery models by recognizing that not all values are necessarily conflicting and identifies a partial order among the values of an attribute using the “subClassOf” and “partOf” relationships. To do this, they use belief functions capable of modeling ignorance and uncertainty and allowing the incorporation of knowledge about the relations between values. Thus, in this modeling several true values can coexist but at different levels of specificity. Consequently, all facts more generic than a given fact are considered true, while conflicting facts are those at the same hierarchical level but with different values.

LFC [131] also measures the reliability of each source (e.g., annotators in the paper) by their specificity and sensitivity relative to the unknown gold standard dataset, assigning higher weights to the best-performing sources. Estimations are iteratively performed using the EM algorithm, where the missing data is the true value initialized by applying a majority voting. The specificity and sensitivity are then estimated iteratively until convergence is reached.

LCA [126] includes four models with different sophistication: SimpleLCA, GuessLCA, MistakeLCA, and LieLCA. LCA is a probabilistic model in which the true value is represented as a multinomial latent variable. To infer the truth, the EM algorithm computes the trustworthiness of each source relative to its claims, after which the true value is determined based on the trustworthiness of the sources.

KBT [38] focuses on assessing the quality of Web sources from which facts are both extracted and evaluated. Facts are extracted as triples (subject, predicate, object) from Web pages using Knowledge Vault, which is composed of 16 different extractors. The authors extend the ACCU model by improving the estimation of source reliability, distinguishing between errors arising from the facts themselves and those resulting from their extractions. However, they do not consider specificity or the possibility that multiple correct values may coexist for a single data item.

While other approaches are focused on categorical data, GTM [180] (Gaussian Truth Model) tackles the truth finding task on numerical data. This model focuses on the relative position of numerical claim values $v_c$ in terms of distance to find the truth. To embed the notion of distance in their model, the authors treat the truth of each entity as a random variable and use it as the mean parameter in the probabilistic distribution for each claimed value of the observed entities. To do this, they leverage a Gaussian distribution for its ability to model errors, thanks to its quadratic penalty. Regarding the evaluation of the quality of data sources, they assume that the quality correlates with the proximity of the claims to the truth. Therefore, the quality of a source is modeled by the variance of the Gaussian distribution, e.g., a high-quality source is represented by a low variance. As aforementioned, the model can take as input the output of another truth finding method or a basic truth estimate (e.g., mean or median value) to mitigate the influence of outliers on the maximum likelihood estimate (MLE). Source quality is derived from a prior inverse Gamma distribution, while the truth for an entity is inferred from a prior Gaussian distribution. Finally, they use the EM algorithm to compute both the truth and source quality.

In the same manner as GTM, LTM [181] incorporates prior knowledge about sources or truth into the truth finding process and introduces the notion of two-sided source quality. It simultaneously infers the quality of the source and the truth, as both mutually influence each other. To assess the quality of data sources, the authors model each source as a classifier, with its own confusion matrix. Thus, the quality of a source is defined by its sensitivity (or recall), which corresponds to the “false negative rate”, and its specificity, which corresponds to the “false positive rate”. These are two independent measures. Both sensitivity and specificity are modeled using a Beta distribution. For specificity, the parameters are “the prior false positive count” and “the prior true negative count”, while for sensitivity they are “the prior true positive count” and “the prior false negative count”. The prior probability of truth is also modeled using a Beta distribution with “the prior true count” and “the prior false count” as parameters for each distinct (entity, value) pair. The truth value is modeled using a Bernoulli distribution with parameter $\theta$, representing the prior probability of the value being true. Finally, the truth and the quality of sources are inferred using Collapsed Gibbs Sampling.

Record Fusion [64] merges knowledge by leveraging integrity constraints, quantitative statistics, and provenance information when available. To find the true value of each table cell, it employs one classifier per column (attribute) in the dataset. These softmax classifiers can be modular, for example a logistic regression, a decision tree, a neural network, and others. Three representation models are explored to create the feature vector, which will then be provided to the classifier. The first representation concerns the role of a cell at the column level, with three proposed strategies. The second representation concerns the role of the cell at the row level (tuple), i.e., it captures the relationship of the attribute with other attributes in its row (entity). Two signals are leveraged, the first one includes the counts of pairs of attributes that are seen together, and the second one captures how often a cell occurs among rows within its own entity. Finally, the third concerns the role of the cell in relation to the complete dataset (table), i.e., takes into consideration the number of denial constraint violations, includes the source information only if available since entities can have different provenance and each source can have different levels of trust. Then, the last step consists in training the different classifiers by a stage-wise additive model for a number of $T$ iterations: (1) they learn the softmax classifiers with the original dataset, (2) use previous predictions to construct a new dynamic feature, (3) and learn again the classifiers using these new sets of features. For cells where the label is unknown, they assign a majority vote as their weak labels.

In contrast to many fusion truth inferring approaches, FaitCrowd [103] evaluates the quality of data sources over several degrees, with one quality degree for each knowledge topic for a crowdsourcing case. FaitCrowd models the expertise of each source for each topic using a Gaussian distribution. It then models the true value provided by a source for a specific question on a given topic as a logistic function depending on the contribution ratio of the source on the topic, its expertise, and a bias term. To estimate the parameters, the model employs the Gibbs-EM inference method, which alternates between Gibbs sampling and gradient descent.

TKGC [72] takes advantage of prior knowledge from the KG when building it and considers that the noise that affects the truth is modeled by a probability distribution specific to each data source. To estimate the difference between the true value and the value provided by a source, the authors use a difference function adapted to each data type, i.e., categorical, numerical, datetime, and string data. This difference function operates on representation vectors previously learned in a fact scoring function for KG completion. The output of this function follows a Gaussian distribution $𝒩(0,(k_a{\sigma }_s^2)$, where $k_a$ is a regularization factor and ${\sigma }_s$ represents the noise of the data source. Finally, TI is conducted using a semi-supervised algorithm.

OKELE [17] models the probability of a fact being true as a latent random variable following a Beta$({\beta }_0,{\beta }_1)$ distribution, where ${\beta }_1$ represents the prior true count of the fact, and ${\beta }_0$ represents its prior false count. The quality of a data source is characterized by its error variance ${\omega }_s$, which follows a scaled inverse chi-squared distribution Scale-inv-${\chi }^2(v_s,{\tau }_s^2)$, where $v_s$ denotes the number of facts provided by the source, and ${\tau }_s^2$ represents the variance. The authors argue that this distribution effectively handles the effect of dataset size, particularly for long-tail entities. TI is performed by leveraging prior knowledge from existing KGs to identify whether an attribute expects a single or multiple values.

HYBRID [92] addresses the TI task for knowledge in tail verticals, i.e., less popular domains that are under-represented. The approach explores data fusion under two assumptions: single-truth and multi-truth. Before applying data fusion, they collect “evidences” on entities by checking whether a source contains the subject and object of the original triple. To do this, they use three types of sources: KBs (Freebase and Knowledge Vault), the Web, and query logs. Provenance information such as the URL where the system found the evidence and the pattern, is retained. Once the evidence retrieval is complete, HYBRID leverages the number of truths for each type of data item as a prior probability (for example, a cell phone has only one year of creation, or between two and eight buttons could be considered for a cell phone). Therefore, when a single true value or multiple true values are expected, it applies a single-truth model or a multi-truth model respectively. To assess the quality of data sources, two metrics are used: precision, i.e., the probability that when a source provides a value, a truth exists and recall, i.e., when a truth exists, the source provides a value.

CATD [93] addresses the problem of data fusion in a context where sources provide only a few claims, making it difficult to estimate their quality. The quality of sources is modeled by a Gaussian distribution, where the mean represents the bias of the source, i.e., its tendency to provide false information intentionally, and the variance represents the degree of reliability of the source. To cope with the problem of limited data from a source, they use a confidence interval of the variance to represent its reliability. Finally, CATD applies an optimization algorithm by initializing the true values with a simple method (e.g., a median of the values). It first estimates the quality of the sources based on their claimed values, then iteratively refines the true values.

KDEm [157] replaces the concept of the true value with the concept of trustworthy opinions regarding the value of an entity. This model allows multiple true values for an entity’s attribute, and consequently considers the completeness aspect of knowledge specificity. KDEm leverages kernel density estimation with a Gaussian kernel and extends it by incorporating the weights of the sources to estimate the probability distributions of values for each attribute of an entity. To find the true values, it combines the density estimation with a threshold and detects outliers that are below this threshold.

An ontology entails three key notions, namely conceptualization, explicit and formal specification, and sharing [54, 53]. The conceptualization refers to an abstract view of a domain, including the relevant concepts and entities, and relates them together. In this way, ontologies make domain knowledge understandable by the machine and enable reasoning about knowledge by defining rules, constraints, and the domain and range of relations [173]. [1] provides a table that summarizes the usual components that form an ontology (refer to this table for further details). OWL allows the description of an ontology of a knowledge domain through individuals, classes or concepts, and properties. It is based on description logic and is part of the W3C’s recommendations. Despite their ability to define rich ontologies, OWL, and more generally, DLs cannot natively represent or reason about knowledge uncertainty, as they rely on crisp logic, i.e., a statement is either true or false unlike fuzzy logic [30]. However, in [27], the authors argue that the inability to handle uncertain information affects the requirements of the Semantic Web. To model and handle uncertainty in an ontology by including their uncertainty theory, most approaches extend the OWL and DLs [109]. These uncertainty theories include the probability theory, for example, with Bayesian network, fuzzy logic, belief functions with Dempster-Shafer (DS) theory, etc.

In [102], the authors describe the different logics available for managing uncertainty, including probabilistic logic, possibilistic logic, and many-valued logic. $ℬℰℒ$ [21] extends $ℰℒ$ to handle uncertain knowledge with the following assumption: the KB contains information that is certain but depends on an uncertain context. One of the advantages of Bayesian networks is their capacity to represent probabilistic dependencies among different elements of a KB [21]. The Bayesian network models the probability distribution of the contexts. Each random variable in the network represents a characteristic associated with these contexts.

In [13], the authors propose a probabilistic extension of the classical DL $𝒜ℒ𝒞$ [3], called $ℬ𝒜ℒ𝒞$. The specificity of $ℬ𝒜ℒ𝒞$ is that axioms and assertions are annotated with an optional context, which indicates the condition for the assertion or axiom to hold. These contexts are associated with probabilities represented in a Bayesian network. Since contexts are optional, classical axioms or assertions can be encoded such that an $𝒜ℒ𝒞$ KB is also a $ℬ𝒜ℒ𝒞$ KB, with axioms and assertions that always hold.

Log-linear DLs integrate logics with probabilistic log-linear models and allow the incorporation of probabilistic and deterministic dependencies between description logic axioms [119, 118]. In [119], the authors present a log-linear DL based on $ℰ{ℒ}^{++}\text{-LL}$ (i.e., $ℰ{ℒ}^{++}$ [87] without nominals and concrete domains). An $ℰ{ℒ}^{++}\text{-LL}$ ontology is a pair (${𝒞}^{\mathrm{D}},{𝒞}^{\mathrm{U}}$), where ${𝒞}^{\mathrm{D}}$ is the set of deterministic axioms and ${𝒞}^{\mathrm{U}}$ is the set of axioms associated with a weight $w$ representing the degree of confidence. These weights determine the log-linear probability distribution.

f-OWL [143] extends OWL DL with fuzzy set theory by adding degrees to OWL facts to represent vague knowledge. The semantics of f-OWL are provided by a fuzzy interpretation, where the interpretation function maps the elements of the domain to a membership function in $[0,1]$, representing the degree.

OntoBayes [173] integrates Bayesian networks into OWL to leverage the advantages of both. It introduces three OWL classes: PriorProb, CondProb, and FullProbDist of type ProbValue (value between 0 and 1) to manage probabilities.

PR-OWL [27] aims to provide a probabilistic extension of OWL since the probability theory can represent uncertainty by combining Bayesian probability theory with First-Order Logic. In addition to OWL, the ontology includes the statistical regularities that characterize the knowledge domain, the knowledge that is incomplete, inconclusive, ambiguous, unreliable and dissonant, then the uncertainty associated with this knowledge. It has the ability to perform probabilistic reasoning with incomplete or uncertain information conveyed through an ontology but requires RDF Reification (presented in Section 9.2) as a probabilistic model includes more than one individual (N-ary relations).

BayesOWL [30] extends OWL to represent and reason with uncertainty using Bayesian networks. The BayesOWl model includes a set of structural translation rules to convert an OWL ontology into a directed acyclic graph of a Bayesian network. It provides the encoding of two types of probability, namely priors and pairwise conditionals through two defined OWL classes PriorProb and CondProb. A prior probability for a concept is defined as an instance of the class PriorProb with two properties: hasVariable and hasProbValue. A conditional probability is represented as an instance of class CondProb with the same properties as the above instance and a property hasCondition.

URW3-XG [88] provides an ontology as a starting point to be refined. A sentence about the world is asserted by an agent. The uncertainty of a sentence has a relation hasUncertainty with a derivationType, uncertaintyType, UncertaintyModel, and a nature. UncertaintyType includes the ambiguity, empirical uncertainty, randomness, vagueness, inconsistency and incompleteness. UncertaintyModel includes probability, fuzzy logic, belief functions, rough sets, and other mathematical models for reasoning under uncertainty. UncertaintyNature categorizes uncertainty as either aleatoric, i.e., ontic, or epistemic.

Poss-OWL 2 [4] extends OWL 2 to represent incomplete and uncertain knowledge from a possibilistic perspective. The ontology has three main classes: concept, role, and axiom. Concept is the equivalent of concept constructor of OWL 2 with an added degree that stands for the certainty level of the concept. Role represents the properties of objects and data. Axiom corresponds to the possibilistic axioms (PossTBoxAxiom and PossABoxAxiom), where each axiom is associated with a real value representing the certainty level of the axiom. The main limitation of Poss-OWL 2 is that it focuses only on the description of uncertainty at the class level.

Riali et al. [134] propose a probabilistic extension of fuzzy ontologies to model vague, imprecise, probabilistic knowledge as fuzzy OWL only models vagueness. Riali et al. also provide a comparison of different approaches for modeling uncertainty in an ontology such as PODM [66], HyProb-Ontology [113], etc.

mUnc [34] aims to unify the different uncertainty theories within a single ontology. The ontology includes the following theories: probability, Dempster-Shafer evidence theory, and possibility theory. mUnc allows publishing uncertainty theories alongside their features and computation methods. Each uncertainty theory is linked to a set of features and operators. The features correspond to the metrics on which uncertainty theory is based to indicate the degree of truth, credibility, or the likelihood of a sentence.

The basic RDF model cannot natively inject values directly into the edges. However, alternative graph representations can circumvent this limitation. To detail these graph representations, we consider that we want to represent the uncertainty through a confidence score $s\in [0,1]$, where 0 indicates low confidence and 1 indicates high confidence. In [2], the authors use 10 criteria to compare five data representation models: RDF, RDF*, Named Graph, Property Graph and their model Multilayer Graph. Among these 10 criteria, we consider that two main criteria are required to represent the confidence score and the provenance of a RDF triple. The first criterion is edge annotation and refers to the ability of the representation model to assign attribute-value pairs to an edge. The second one is edge as nodes, meaning that an edge can be referenced as multiple nodes. Therefore, we review the data representation models w.r.t. these two criteria and their pros and cons. We illustrate the models with the triple <JoeBiden, isPresident, UnitedStates> associated with the confidence score “0.911” in Figure 11.

Singleton Property [116] uses a new type of property called “singleton property”, which corresponds to a unique property linked to a URI between two entities. This unique property can act as a node to which additional relations can be added. For example, the singleton property in Figure 11(b) is “isPresident#1”, then this node is used to add the confidence value “0.911”. Despite the fact that the singleton property is convenient for a compact meta-level representation, this modeling introduces many unique predicates and affects data querying [135, 61].

Property Graph puts additional information about triples that are stored as a list of key/value pairs at edges in the graph. For example, in Figure 11(c), the confidence value and the provenance are attached to the relation “:isPresident”.

Named Graph [65] extends the RDF triple model and allows to indicate a triple as a subgraph denoted by an IRI. This subgraph with an identifier can be used to add meta-information. In Figure 11(d), the original RDF triple <JoeBiden, :isPresident, UnitedStates> is identified by “:statementId#1”, which is used as the subject in the triple <:statementId#1, :confidenceValue, 0.911>. This modeling, which corresponds to nested graphs, is well-supported in the SPARQL standard and well-suited for representing provenance data [135].

N-ary [104] creates a node to represent a relation concept whose triples linked to this node correspond to the arguments of the relation. For example, in Figure 11(e), an intermediate node “President1” of type “President” characterizes the relation “:isPresident”, then meta-data can annotate the relation. The main drawback of this representation is its cumbersome syntax, which increases the complexity of the KG since the n-ary relation must be divided into several binary relations.

RDF reification [62] involves creating an Internationalized Resource Identifier (IRI) or blank node that plays the role of the subject of all triples, as depicted in Figure 11(f). To represent the former triple it uses three new relations namely rdf:subject, rdf:predicate, and rdf:object then as many relations as it needs to add metadata. This modeling was the first way to make statements about statements [135]. This method is simple, but its syntax is too verbose since each statement must be reified. This considerably increases the size of the KG and complicates both SPARQL querying and RDF data exchange [116, 2, 135, 60].

RDF-star [61] is an extension of the RDF model proposed by the Semantic Web community. A RDF triple is a tuple $t^{\ast }\in (T^{\ast }\times E)\times R\times (T^{\ast }\times E\times L)\in T^{\ast }$ and RDF-star triple is a tuple $t^{\ast }\in (T^{\ast }\times E)\times R\times (T^{\ast }\times E\times L)\in T^{\ast }$ where $E$ is the set of entities, $R$ is the set of relations, $L$ the set of literals, and $T^{\ast }$ is the set of RDF-star triples. RDF-star can expressively extend an existing RDF model, since the triple metadata are simply added as objects of them [80]. For example, in Figure 11(g), metadata such as provenance or confidence scores are added directly to the triple <BarackObama, :isPresident, UnitedStates> illustrated by . In addition to the ability to add metadata at the statement level without modifying the remaining data, RDF-star has its own query language called SPARQL-star which reduces compatibility issues [84]. A comparison conducted on Wikidata demonstrates that RDF-star performs better than reification, n-ary, and named graph representations in terms of the number of triples, loading time, and storage capacity [80]. However, RDF-star cannot consistently represent the same metadata with different values for the same triple [60, 2]. Indeed, in Figure 11(g), there are different start dates that refer to the same triple.

Multilayer Graph [2] unifies the various advantages of the other representations we have described so far into a single, simple, and flexible (whether at node or statement level) model by introducing the notion of “layer”. To explain this, we use the notations and definition of a multilayer graph from [2]: given $Obj$ a universe of objects that contains strings, numbers, IRIs and so on, a multilayer graph is defined as $G=(O,\gamma )$ where $O\subseteq Obj$ is a set of objects and $\gamma :O\to O\times O\times O$ is a partial mapping that models directed, labeled and identified edges between objects. The layers in the multilayer graph arise from the nested structure of edge ids. The layer of an object $o\in O$, described as layer($o$) is defined as follows: if $o$ is not an edge identifier, then layer($o$) = 0; otherwise if $\gamma (o)=(n_1,l,n_2)$ then layer($o$) = max{layer($n_1$), layer($l$), layer($n_2$)} + 1. Figure 12 depicts the layer representation of (h) from Figure 11(h). This data model allows the unambiguous representation of multiple provenances and varying confidence scores within a single triple.