NEOntometrics – A Public Endpoint for Calculating Ontology Metrics

Lozano-Tello and Gómez-Pérez published the Ontometric framework in 2004. It proposes evaluation attributes in five criteria: (Development) tool, (ontology) language, context, methodology, and cost. Arguably, some metrics have become obsolete due to standardization in the past years. In 2004, the web ontology language (OWL) was just released, and other representations like OIL, DAML+OIL, and SHOE were still actively used. Here, Ontometric is targeted to make the influences of the languages explicit and comparable [13]. Today though, regarding the category languages, RDFS-based ontologies can be considered state-of-the-art and are mostly compatible with each other. Further, the standardization decoupled the tools from the ontology. Thus, the tool capabilities do not influence the semantic artifact. Other proposed elements, however, can be supported by an automatic calculation, like the metric maximum depth in the category content. While the relevance today might be somewhat limited, Ontometric considerably influenced the newer frameworks.

Gangemi et al. proposed an ontology evaluation framework based on a semiotic meta-ontology O² that provides a formal definition for ontologies and their usage. Further, the authors define an ontology evaluation design pattern (oQual). Based on their O² definition, measurements assessing structural, task, corpus, and usability-based attributes are proposed [8]. A technical report by the same authors further suggested 32 metrics in seven categories assessing mostly graph-related structures like depth, width, modularity, the distribution of siblings, or tangledness [9].

In 2005, Burton-Jones et al. presented the semiotic metric suite. It comprises four main categories (syntactic, semantic, pragmatic, and social quality) and ten quality metrics. While some of these metrics are based solely on the structure of the ontology itself, others need further additional external information. Nine of these measurements, in theory, can be calculated automatically [2]. Practically, some of the required data for some measures will probably not be available. Examples are the access count of the ontology or the links from other ontologies to the currently assessed one.

OQuaRE was first proposed in 2011 by Duque-Ramos et al. It has since been used in several publications, always involving the core team of the proposing authors. OQuaRE offers 19 calculatable metrics and associates these metrics with quality dimensions like readability or accountability. Further, the framework ties metric results to quality ratings, thus providing an interpretation of the measurements. This holistic approach to quality is a unique characteristic among the frameworks [6]. However, during implementation, we experienced several heterogeneities in the metric definitions of the framework, with metrics having the same name, created by the same authors, being defined differently in their publications. To facilitate further research on the framework and to integrate the framework into NEOntometrics, we reworked the OQuaRE measurements. However, empirical research with the newly implemented metrics showed that their proposed linkages from measures to ontology quality scores do not sufficiently work [27]. Our study was the first made by authors not part of the team that proposed the framework; however, the NEOntometrics applications shall allow more thorough analysis in the future.

Tartir et al. published 19 metrics in the OntoQA framework in 2005 and 2007. While the framework does not provide a grading system for metrics like OQuaRE, it aids the interpretation by describing how modeling decisions influence the metric results. Further, the authors propose measurements applicable not to the ontology as a whole but to the elements in an ontology. OntoQA also defines class- and relation-specific measurements. The relationship importance, for example, is calculated once for each relationship [32, 31].

There are further metric frameworks that consider the cohesion of an ontology. Yao et al. propose a set of measures based on an inheritance tree [35]. In a consecutive paper, the authors further provide an empirical analysis and interpretation context [17]. Ma et al. examined the ontology partitions with special consideration of the open world assumption [14].

Over the years, a lot of frameworks have been proposed. As Table 1 shows, these papers have gathered many citations over the years. Some of these frameworks are merely theoretical in their proposals; others came with prototypical implementations. Further, tools that do not correspond to one of the proposed frameworks have been developed.

The following section shows historical and current software for ontology metric calculation.

OntoKBEval by Lu and Haarslev [19] analyzes the structure of ontologies by providing graph-related measures like the number of levels or the number of concepts per level. The tool offers means to grasp clusters in the ontology and developed its own visualization “Xmas”-tree.

Tartir et al. developed a standalone java application for the OntoQA framework [32], implementing measures of the OntoQA framework, including metrics for the individual classes.

OntoCat, proposed by Cross and Pal [3], is a plugin for the Protégé editor and provides size- and structure-related metrics. They allow the assessment of the ontology as a whole but also provide metrics concerned with specific subsets of the given ontology.

S-OntoEval by Dividino et al. [4] serves as a calculation tool for, among others, the framework of Gangemi et al. Its primary focus is on structural evaluation. However, the tool also calculates usability based on annotations and task performance based on ontology querying.

The Protégé editor [16] offers basic metrics on its landing page that counts the usage of OWL-specific language constructs like the number of object property domain declarations or the number of classes.

The developers of the OQuaRE framework introduced a web tool to calculate their proposed metrics. It integrates a statistical correlation analysis of the metrics and a web service. The tool suffers from the same issues as the framework [27], and the implemented metrics are heterogeneous and do not adhere to a clear definition.

Amith et al. developed the Semiotic-based Evaluation Management System (SEMS), later renamed OntoKeeper [1], which implements the semiotic suite by Burton-Jones et al.

The “OntOlogy Pitfall Scanner” (OOPS) by Poveda-Villalón et al. [18] approaches automatic ontology evaluation differently: They detect common modeling pitfalls like the use of is relationship instead of rdfs:subClassOf or wrongful equivalent relations.

OntoMetrics, first developed by Lantow [11], is a web service for calculating several ontology metrics. It covers most of the OntoQA and oQual ontology metrics and integrates the OWL-based axiom counts that are also part of Protégé. It was later extended with a web service by Reiz et al. [22].

As the previous section has shown, many frameworks and tools have been developed over the past years. That raises the question of whether a new calculation tool is necessary. We argue that our application fills essential gaps:

Missing Practicality.

As Table 2 shows, most of the developed tools are no longer available. Even if they are available, their usability is often low. Many of the tools were used for the authors’ evaluation efforts and do not come with a state-of-the-art user interface. Further, most of the software is not maintained. This problem is amplified by the fact that most of the software is:

Closed Source.

None of the evaluated tools is fully open source (cf. Table 2). Not only hinders this reproducibility. It also prevents the community from maintaining the software and building on this previous research. If there is a need for another kind of evaluation, one has to start from scratch. We, thus, argue that the closed source leads to:

Tool	Date	Type	Available	Open Source	Ref
Onto-KBEval	2006	S	No	No	[19]
OntoQA	2005	S	Yes	(No)222https://github.com/Samir-Tartir/OntoQA.Thebinary.jarfilesareavailableunderCClicense.Thesourcecodeitselfisnotpublic.	[32]
OntoCat	2006	P+	No	No	[3]
S-Onto-Eval	2008	S	No	No	[4]
Protégé	2015	S	Yes	Yes	[16]
OQuaRE	2018	WT, API	Yes333http://sele.inf.um.es/ontology-metrics	No	[27]
Onto-Keeper	2017	WT	No	No	[1]
OOPS	2012	WT, API	Yes444http://oops.linkeddata.es	No	[18]
OntoMetrics	2015	WT, API	Yes555https://ontometrics.informatik.uni-rostock.de,opi.informatik.uni-rostock.de	No	[22, 11]

Isolation.

The implementation efforts have stayed mainly isolated from one another. Hardly any tool has reached a broad acceptance within the community, and the ontology evaluation efforts of researchers using different tools are often not comparable. While there is a consensus that ontology evaluation is meaningful, there is no common understanding of how to do it.

One design goal was to create a flexible application for integrating new metrics. A researcher shall be able to adapt the application to their needs and quickly implement new required metrics.

To achieve this adaptability, we did not encode all of the metrics of the various frameworks directly in the software but decomposed them into their building blocks. For example, the metric Axiom/Class Ratio is not calculated during the ontology analysis. Instead, their building blocks Axioms and Classes are analyzed and saved in the database. The compositional values are then calculated at the time of querying.

The information on the ontology metrics is stored in an OWL-based metric ontology¹³¹³13https://raw.githubusercontent.com/achiminator/NEOntometrics/master/Git-Extension/rest/metricOntology/OntologyMetrics.owl. On startup of the application, multiple SPARQL-queries extract the codified knowledge and set up the backend and frontend. Thus, changing and adapting the ontology is sufficient to adapt the measures. The work [26] further details the underlying metric ontology.

Figure 1 presents an example of the metric elements in the ontology. Elemental Metrics contain the atomic measures that are used to build the compositions. For Axiom Class Ratio, the Elemental Metrics are Axioms (the number of axioms) and Classes (the number of classes). The ontology further specifies mathematical relationships between the metrics. In the given example, Axiom Class Ratio is the subClassOf (divisor only Classes) and (numerator only Axioms). The Elemental Metrics are connected to metric instances named identically to the implementation names in the calculation service and the elements in the database. In the example of the Axioms, this element has a relationship implementedBy value axioms.

All elements have rich annotations, providing human-centered meaning to the metrics. Some elements have links to further online resources or scientific publications. The annotations are the foundation for the Metric Explorer, where users find guidance on the available metrics.

New metrics that build upon the available Elemental Metrics can be created by modeling them in the ontology. Upon start, the application will make these custom metrics automatically available in the front- and backend. Table 4 shows the frameworks that are already implemented in NEOntometrics and part of the Metric Explorer and the Calculation Unit at the time of publication. The case study in Section 4.2 details how to create new frameworks, e.g., for individual metric frameworks in an organization.

The application is based on a dockerized microservice architecture and consists of five components: the calculation-unit OPI (Ontology Programming Interface), the API, the worker application, a database for storing the calculated metrics, and a Redis interface for queueing jobs. The API and worker share a common codebase. Figure 3 depicts the interaction of the involved services.

The frontend contains the GUI. It is written using the multi-platform UI language flutter with its underlying client language dart⁸⁸8https://flutter.dev/, https://dart.dev/. Upon loading, the frontend first queries the API for available ontology metrics based on the metric ontology. This data fills the help section Metric Explorer, which allows users to inform themselves about the various available metrics and the options for the calculation page. Afterward, the user can retrieve the requested ontology metrics or put them into the queue if they do not yet exist.

The API is the django-based⁹⁹9https://www.djangoproject.com/, https://www.django-rest-framework.org/ endpoint for accessing already calculated metric data or requesting the analysis of new repositories. During the startup of the software, the application queries the metric ontology. It builds the frontend data and dynamically creates calculation code to provide the measurements of the frameworks that build upon the Elemental Metrics. After startup, a client can exploit GraphQL to check whether the data he requests exists already in the database. If so, he is able to retrieve all the selected metrics for a given repository. If not, it is possible to put the calculation of a given repository in the queue and track its progress.

The worker is responsible for the calculation of the metrics itself. It checks whether jobs are available in the scheduler Redis database. If that is the case, it starts the analysis by first cloning or pulling the GIT-repository, then iterating through every new file and commit, analyzing the ontologies using the OPI metrics endpoint. Afterward, the calculation results are stored in the database. The scheduling mechanism is based on django-rq¹⁰¹⁰10https://python-rq.org/patterns/django/. Even though the worker shares a code base with the API, it runs as a separate application. The number of parallel calculations can be scaled by increasing the number of workers.

The calculation service OPI is responsible for calculating metrics out of ontology documents. While it is based on the calculation service published in [22], most underlying code has been replaced. The old application struggled with ontology files larger than 10 MB due to inefficient memory allocation and had no separation of the calculation of the elemental metrics and the composed metrics of the metric frameworks. The old application was designed as standalone software, while the new calculation engine is hidden from the user and only accessed by the API.

The backend utilizes two languages: The API is written in python, and the calculation service OPI builds on Java. While the two languages add complexity to the application design, they allow the use of established frameworks for their given tasks.

The calculation and retrieval process as a whole is depicted in Figure 2. At first, the frontend requests whether an ontology is already known in the system. If not, it either returns the queue information to the end-user or starts another request for the ontology metrics. At the same time, the worker applications and OPI handle the queued tasks.

The microservice architecture allows utilizing the strengths of the various languages. The Java calculation unit builds on the OWL-API for an efficient graph traversal. The Python/django application is easily extensible through the availability of multiple plugins, e.g., for the GraphQL endpoint and the asynchronous metric calculation. The flutter-based frontend comes with built-in material design support. Further, the microservices allow for potential horizontal scaling of the calculation.

The page Metric Explorer is a dynamic help page of available metrics in NEOntometrics. The two main categories are Elemental Metrics and Quality Frameworks. The former contains the underlying atomic measurements of the ontologies. The authors of NEOntometrics create all the information shown in this category. Quality Frameworks, on the other hand, present the ontology metrics developed by other researchers, like the OntoQA Metrics [32] by Tartir et al., shown in Section 2.1. Here, all information originates from the authors of the given frameworks.

The Metric Explorer provides information on five categories (though not all are filled for all the metrics). Metric Definition contains the formal definition of the metrics and how they are calculated, while Metric Description supplements a more human-readable explanation and, at times, an example. The Metric Interpretation guides practical usage. Calculation explains their decomposition into the Elemental Metrics using the metric names that are returned by the API, and seeAlso links to further resources like the corresponding papers or additional reads.

The interactive help page is closely bound to the metric ontology. Every change will be reflected in the Metric Explorer after a restart, and the ontology provides annotation properties for the metric definitions, descriptions, and interpretations. The nesting of the measurements is defined by the subclass relations in the ontology, and the calculation field is defined by the object properties representing the mathematical relationships (cf. Figure 1).

For direct consumption by a user, the tab Calculation Engine (as shown in Figure 5) is the main entry point for the metric calculation. The end-user first selects the required metrics. The “Already Calculated” button shows the calculated repositories already stored in the database. While these repositories can be a starting point for further exploration, the user can also place a URL in the textbox that points to a new public GIT repository or the location of an ontology file.

A click on the arrow starts the metric request. Once the data is analyzed, clicking the arrow leads to the metric results presented as a paginated table, representing the metric values for the different ontology versions. A drop-down menu in the header allows selecting the various ontology files, and the download button exports the metrics into a .csv. A click on the button “show the analytic” opens internal visualizations for displaying the ontology evolution, comparing the various ontologies in a repository, and assessing the recent changes.

One goal of NEOntometrics is to allow the integration of ontology metrics into semantic web applications, which requires exposing the service using a standardized interface. Relevant open standards are REST and GraphQL for the web and SPARQL for querying the semantic web. NEOntometrics builds on GraphQL.

GraphQL (together with REST) has become a new de facto standard for sharing information on the web, and there is broad support in various programming languages and frameworks. This support includes the django web framework used in this project, where the graphene plugin¹¹¹¹11https://graphene-python.org/ allows utilizing the internal Object Relational Mapping. It allowed us to build the interface with comparatively little implementation effort, as the requests are translated to database queries automatically. While these integrations are also available for REST, GraphQL allows for the traversing of relationships and the precise selection of attributes for querying. Avoiding over-fetching is highly relevant for this use case, as one ontology version has over 100 ontology metrics, and the user likely selects just a few.

The GraphQL endpoint further provides documentation on the various available requests and possible return values, thus enabling the guided development of new queries. The interface is accessible through a browser on a GraphiQL interface or any other GraphQL client.

SPARQL, as a graph-based query language, has similar attributes to GraphQL regarding relationship traversal and attribute selection. Additionally, proving a SPARQL endpoint would further allow the integration of the metric calculations into existing knowledge bases. Unfortunately, there is (currently) no support in the form of plugins for integrating such an endpoint into the used django framework. This lack makes the creation of such an endpoint dissimilar costlier.

Analyzing ontology metrics over time can tell a lot about underlying design decisions. The size of the changes indicates if an ontology evolves gradually or has disruptive changes, thus measuring stability and identifying the disruptive changes. They also allow us to assess how attributes like the logical complexity (e.g., measures through the number of axioms that incorporate meaning), the coverage (e.g., measured through the number of classes or individuals), or the shape of the graph (e.g., measured through depth or breadth) change over time.

This case study analyzes the Evidence and Conclusion Ontology (ECO). ECO captures the biological coherences like “gene product X has function Y as supported by evidence Z” [10]. The NEOntometrics authors have no affiliation with the authors of the ontology nor with the biomedical field of research. Further, the goal of the section is not to evaluate quality but to observe the development of the ontology over time to give an impression of possible assessments. Previous work discussed the connection between metrics and development decisions from an ontology engineering perspective [24, 36].

The ECO repository has 856 commits in 17 ontology files. For this analysis, we were interested in the main ontologies in this repository. Thus, we only assessed the ontologies in the root structure, resulting in three ontology files. eco.owl with 89 versions, eco-basic.owl with 44, and eco-base.owl with 45 versions. We first examined the axiom count of the ontologies and then used Tartir et al.’s OntoQA framework [32, 31] for further analysis. The corresponding source files in Jupyter Notebooks are available online¹²¹²12doi.org/10.5281/zenodo.14047141.

The first analysis is concerned with the development of the ontology size. Figure 6 presents the three ontology files in their different versions and plots the development of axioms with time. While the solid line represents all axioms overall, the dashed line only accounts for such that incorporate a logical meaning in RDFS or OWL syntax. The difference between the dashed and the solid lines are, thus, annotations or custom-defined properties.

An insight of the chart in Figure 6 is the variances of the logical axioms and the axioms in general. While the size of the ontology overall fluctuates intensely, the number of parts of the ontology that incorporate logical meaning stays relatively stable. One significant spike occurred between 2018 and 2019, which we will scrutinize further. Analysis reveals that a more extensive restructuring of the ontology drives this increase in logical axioms. The classes in eco doubled from around 900 to a little over 2,000, then increased to over 3,000. The number of defined object properties jumps from three to above 50, and the relation on classes through object properties increases from 350 to almost 2,500, then drops to around 1,600. This change event also marked the introduction of eco-base and eco-basic.

Figure 7 shows the relationship richness and schema deepness defined in the OntoQA framework. The former is defined as the number of non-inheritance relationships divided by the sum of non-inheritance relationships and inheritance relationships, and the latter is the number of subclasses per class [32].

Figure 7 (left) shows that, after an initial increase due to the rise in object properties, the relationship richness of eco drops with the increase in classes and subClassOf statements. Also, object properties were introduced. Later, a decline in object properties, combined with the further increase in classes and subClassOf statements, partially reverses the growth.

The right diagram in Figure 7 provides more insights into the role of sub-class relationships. At first, a lot more subClassOf relationships than classes were introduced. However, the number of subClassOf relations later stagnated, even getting smaller. In contrast, the number of classes increased steadily. This suggests that the rebound in the relationship richness is driven more by the decline in object properties than the increase in subClassOf relationships. While the number of logical axioms is more or less stable, the underlying logical attributes of the ontology that constitute how the ontology is structured are still subjected to changes.

There are many more aspects that one could analyze for the given repository, and this section is merely a short demonstration of the value of environmental metrics. As the last diagram indicates, many more fluctuations are worth looking at. The variations affect the relationships between non-hierarchical and hierarchical relationships, classes, and graph-related structures like the width or depth, individuals, or data properties.

A recent empirical analysis of 69 ontology evolution processes (based on NEOntometrics) has shown that the developments are highly heterogeneous and that assumptions on stereotypical development processes do not apply: There is no common rule or joint history that ontologies share [28]. If the ontologies are highly diverse, so is the required evaluation. This diversity of ontologies and their metrics emphasizes the careful selection of the latter. One person might build a taxonomy with rich human-readable annotations and other targets to infer knowledge by modeling complex class relationships. A successfully applied metric by the first person might not work for the second. While the Metric Explorer supports the selection process, a person might develop their measure to intertwine two metrics in a way that has not been done before. Organizations may want to select and reorder the metrics or limit the display to only relevant ones. The following subsection explains how to adapt the application by altering the metric ontology. While every ontology editor can be utilized for editing, this section builds on the open-source software Protégé [16]. The metric ontology is stored in the GitHub repository¹³¹³13https://raw.githubusercontent.com/achiminator/NEOntometrics/master/Git-Extension/rest/metricOntology/OntologyMetrics.owl

The aims motivating the development of NEOntometrics express the importance of creating an approach that is mature enough to be applied in the ontology engineering community without substantial development efforts. Our assumption is that the more an approach has been evaluated in theory and practice, the more mature and useful it is. Among the many scientific approaches for evaluating research results, we base our evaluation strategy on the work of Lincoln and Guba [12, p. 289 ff.] on “naturalistic inquiry”.

Lincoln and Guba distinguish between theoretical and practical validation. Theoretical validation means assessing an approach within the theories of the domain to which the approach is part or supposed to contribute. In the context of ontology metrics, this means assessing the soundness, feasibility, and consistency within the body of knowledge, such as ontology engineering and knowledge engineering. Practical validation encompasses all kinds of application of the approach for evaluation purposes, which requires defined procedures and documenting results. This could be simple lab examples illustrating the approach, controlled experiments in a lab setting, applications in industrial cases, etc.

Furthermore, Lincoln and Guba also consider the context of validation and distinguish between validation by the approach’s developers in their internal environment, validation by the developers outside the internal environment, and validation by actors other than the developers. Combining these two perspectives leads to a two-by-three matrix, as depicted in Table 5. The cells of this table show typical ways of validation for the different combinations of the two perspectives.

Usually, validation starts on the “internal, development team” level with validation in theory followed by validation in practice, and proceeds “downward” in the matrix with alternating theory and practice validation to “external, in application context”. Thus, the highest validation status would be reached if all cells in the matrix were covered.

As described above, Lincoln and Guba focus on validating research results, i.e., to check whether or not a certain result from research is appropriate for its purpose. In our case, the purpose of NEOntometrics is defined by the objectives (see introduction to this section 5). Validation and evaluation, even though different from each other, are very much linked. Evaluation is the process of assessing (and often computing) key characteristics of the research results, which can be used for validation purposes. As many of the NEOntometrics objectives require measurements instead of only checking characteristics, we use the term evaluation episodes in the next section.

Table 6 shows the performed validation episodes following Lincoln and Guba’s naturalistic inquiry framework. This section aims to summarize the intention and results of the different episodes. To emphasize the importance of the usefulness and applicability of the tool for the ontology engineering community, we put a focus on external validation. Many external validation episodes were published in peer-reviewed publications, and peer-reviewing was also used as an instrument for external validation. In total, six publications contribute to the evaluation steps.

Besides the additional functionality powered by the microservice architecture of NEOntometrics, the calculation engine has also been reworked to improve performance and resource consumption. The following evaluation compares the old OntoMetrics [25] with the performance of the reworked metric calculation (cf. Figure 3). The old Ontometrics engine calculates 72 measurements, 16 of these measurements are ratio-based metrics. The new calculation engine only measures the underlying atomic measures, 83 in total.

For the test, we run both calculation services in dockerized environments on the same machine (Lenovo z13 G2, 64GB Ram, AMD Ryzon Pro 7840U). We run a calculation of three different ontologies: The puppet-Disco inferred ontology¹⁴¹⁴14https://raw.githubusercontent.com/kbarber/puppet-ontologies/master/puppet-disco/inferred.owl, the larger human disease (DOID) ontology with 24MB¹⁵¹⁵15https://raw.githubusercontent.com/DiseaseOntology/HumanDiseaseOntology/main/src/ontology/doid.owl, and the ECO ontology of Section 4.1 with 7.2MB¹⁶¹⁶16https://raw.githubusercontent.com/evidenceontology/evidenceontology/master/eco.owl. Ten times, we send the puppet ontology, then human disease, and finally, the ECO ontology to the new calculation service and then the old one. We measured the response time and queried the docker stats to analyze memory footage. The jupyter notebook used for analysis is available online¹²¹²12doi.org/10.5281/zenodo.14047141. The first analysis presented in Figure 9 displays the average calculation times for each of the ontologies. The small Puppet Ontology is sped up 35 times in its analysis, while the human disease ontology calculates 6.5 times faster, and ECO 29 times faster.

The second analysis in Figure 10 measures the engine’s memory consumption by extracting the docker statistics after each run. Here, the rework of the calculation engine cut the required memory of the analysis to a fifth.