LLM-Supported Manufacturing Mapping Generation

In large manufacturing companies, such as Bosch, data related to a specific machine inventory item is stored in distributed databases, which are built and curated to serve the needs for one or multiple functions within the company, see Figure 1. Cross-functional questions, e.g., linking physical location and financial information, often lead to treasure hunting with laborious human efforts, experts’ alignments, and delays when looking for the answer. While efforts increase with growing data volumes, the pressure on fast and sound decisions in the manufacturing domain rises.

The semantic approach has proven to be successful in bringing heterogeneous data to a common format and interlinking datasets, e.g., tables from relational databases, with the help of ontologies and Knowledge Graphs (KGs). Indeed, it has already been adopted in various industrial sectors such as energy [12], manufacturing [17], e-commerce [1] and others. In particular, Bosch has been relying on ontologies and KGs in many applications ranging from analyses of discrete manufacturing [29] to autonomous driving [22] and combinatorial optimization [5]. Moreover, Bosch has been scaling the use of semantics in various large projects such as the European data infrastructure project Gaia-X²²2https://gaia-x.eu/news-press/the-role-of-ontologies-in-gaia-x/ and its derivative Catena-X.

In these applications the key task is to link a domain ontology to the original data with the help of declarative mappings and then execute the mappings in order to create a domain KG. The creation and maintenance of manufacturing mappings is non-trivial. Especially, domain-specific vocabulary, updates in data and data structure as well as complex facts and relations across different data silos pose challenges. As of now, it is common practice to create them manually. For this, a domain expert and a knowledge expert are required and they align during the mapping generation. Mappings must be conform to the ontology of the KG. The domain expert understands which real object or concept is actually referenced by the label in the data set, whereas the knowledge expert retrieves related ontologies, understands the ontology syntax, and creates syntactically correct mappings.

YARRRML has been introduced as a human-readable representation of RDF mappings and provides an alternative to the standard mapping languages Relational Database to RDF Mapping Language (R2RML)³³3https://www.w3.org/TR/r2rml/, https://www.w3.org/2001/sw/wiki/RDB2RDF and RDF Mapping Language (RML)⁴⁴4https://rml.io/specs/rml/. Van Assche et al. [23] assess in a systematic literature review, eight mapping languages excluding YARRRML, noting that “it was excluded because it is published as a demo”, yet acknowledging its widespread use in practice. We can second the industrial use of YARRRML, e.g., at several plants at Bosch, due to its vendor independence as well as further enhancements (see e.g. [10], [24]). Van Assche et al. [23] assess RML which is an extension of the W3C standard language R2RML from databases only to further structured sources such as .csv or .json. High quality mappings are of core importance for a functional manufacturing KG, yet their creation remains, with both RML or human-friendly YARRRML, challenging. We explore mapping generation on each of the two languages in our work.

LLMs in a manufacturing context often work with unstructured data such as text (e.g., error descriptions) or images (e.g., optical inspection of parts). Research on neuro-symbolic KG construction in manufacturing ([20]) shows that most such approaches are conducted on unstructured data sources. Exploration of neuro-symbolic AI approaches for creating mappings for KG construction in manufacturing is still missing. As LLMs with their well-known strength in generating text, and also code, from unstructured text rapidly evolve, it is promising to explore their capabilities of dealing with structured data, e.g., mapping tasks [11]. However, varying accuracy, token size limitations and missing domain-focus of LLMs require advanced generation workflows and configurations to realize the applicability of this approach. Retrieval-Augmented Generation (RAG) approaches and neuro-symbolic architectures such as semantically context-enhanced prompts can increase accuracy and domain-focus. Additionally, ontology reduction methods show means of reducing the prompt size, while potentially improving accuracy. We explore this in our work. Our guiding question is:

Can LLM-supported and context-enhanced mapping generation support human expert mapping creation for manufacturing KGs?

This work focuses on addressing this question and the contributions of this paper are summarized as follows:

1. 1.

   We provide the first exploratory approach on LLM-supported and context-enhanced mapping generation for KGs as support for knowledge experts in a manufacturing company.

2. 2.

   We propose a novel combination of context-based ontology reduction and a RAG-based approach for automatic mapping generation to create manufacturing KGs.

3. 3.

   We follow a pragmatic evaluation approach for mapping quality on manufacturing KGs and apply our method on several configurations against gold-standard mapping references which are created by domain and technical experts. By this, we obtain comprehensive insights on influencing factors for high quality manufacturing mapping creation.

The further parts of this paper are structured as follows: We introduce the use case and existing challenges in Section 2. In Section 3, we propose our approach for manufacturing mapping generation and explain the experimental implementations. We present our results on LLM-supported and context-enhanced mapping generation and lessons learned in Section 4 and briefly discuss related work in Section 5. We conclude with future work in Section 6.

In this section, we outline the use case and challenges in manufacturing inventory identification. Currently in Bosch, process experts and data analysts have no common shared and accessible data base that provides information about machines executing the same process. Data items for a single machine are spread across different silos. If the relevant information is available and enriched with various other influencing factors, process experts can roll out solutions easier and faster. For instance, to be able to answer the following question: “How many winding machines are there in a certain plant”? data from different sources needs to be integrated. To that end, a KG has been built to semantically integrate and harmonize data to cope with such questions. Next, we list the main competency questions (CQs) for the KG creation and maintenance. The questions are a prioritized excerpt from a list of questions from practice. The questions were originally collected by domain experts for the overall ontology- and graph-based approach and the selected questions address all three manufacturing ontologies from our scenario. For validation of their currentness, the questions have additionally been reviewed by a subject expert. The CQs are:

• $\mathrm{■}$

  CQ 1: How many machines of a specific type are located in a given plant?

• $\mathrm{■}$

  CQ 2: Where is the machine inventory located that is allocated on my cost center?

• $\mathrm{■}$

  CQ 3: Is the insecure machine control component still used in other stations within the plant?

• $\mathrm{■}$

  CQ 4: What are all financial information to a line with only the machine identifier given?

• $\mathrm{■}$

  CQ 5: What is the unique identifier number to a CMDB asset?

Over time, changes are done in the current data set structure and further data sets will need to be added, so that answers to these questions are up-to-date: the KG supports these tasks pragmatically. For this, mapping generation and updates are needed. With growing data volumes and ontology sizes as well as the need for more efficiency, (semi-)automated mapping generation approaches are needed. However, multiple challenges occur in manufacturing mapping generation in practice if mappings are generated automatically, e.g., with the support of LLMs:

• $\mathrm{■}$

  Challenge 1: How to feed a manufacturing ontology into a prompt?

• $\mathrm{■}$

  Challenge 2: Can context-based ontology reduction methods improve mapping generation for manufacturing KGs?

• $\mathrm{■}$

  Challenge 3: Can prompt context enhancements improve the quality of the generated mapping?

• $\mathrm{■}$

  Challenge 4: Which sample size of a data set is needed for a generated mapping of high quality?

Next, we describe the data sources required to answer the main competency questions.

In this section, we introduce MYAM+R, a Manufacturing YARRRML Mapping generation approach, which also support RML mappings. It comprises four input modules, an ontology reduction module, the core mapping module which generates a mapping, an evaluation module and a KG generation module. In Figure 3 we show an overview of MYAM+R. In the following, we explain all modules and process steps. We conclude this section with implementation details of MYAM+R.

In this section, we show our experimental results and evaluate the outcome, both quantitatively and qualitatively. With multiple setting configurations, we address the challenges described in Section 2 to achieve mapping generations with high quality and reduced prompt size. We evaluate the generated mappings against gold standards with F1-score, precision and recall. Lastly, we collect feedback on our approach and outcome. We first describe the gold standard mapping.

In this section, we discuss relevant work on LLM-supported mapping generation in manufacturing. First, we review works on automatic mapping generation for KG creation and their applicability to the manufacturing domain, as well as briefly consider schema-supported mapping generation. Second, we discuss the current state of research on LLM-based KG construction. Our approach of using data input, ontology and instructions is comparable to a Graph RAG architecture as the steps of enriching an LLM prompt with schema information to generate an output, in our case a mapping, are comparable to the components of a Graph RAG. However, there is one major distinction: as our goal is KG construction, we must not assume existing supporting graph instance data in our framework.

The research field of automatic KG construction has been popular for a long time with new recent approaches. Zhong et al. [28] state in their corresponding survey (2024) that “[l]arge pre-trained models have been leading a significant impact on multiple KG construction tasks and their related applications”. Our focus on Mapping Generation corresponds to the step of Knowledge Acquisition which mainly covers Entity Discovery and Relation Extraction for (semi-)structured data sources according to [28] as they also include tables in their analysis (however, without focus on them). The authors state that rule-based extraction methods “are the general solutions for NER” [28], with Named Entity Recognition (NER) being the first step of identifying and extracting entities from a data source as part of Entity Discovery. In the manufacturing domain however, the disadvantage of this approach lies in the challenge of identifying the correct pairs of (i) source column name and (ii) corresponding entity in the ontology as these pairs may differ drastically on a lexicographical level. Hence, experts conduct this decision manually or, for overall manufacturing KG construction, other “advanced deep learning methods such as BiLSTM-CRF, BiLSTM-ALBERT, etc.” [21] are utilized. While experts’ time is rare and costly, deep learning methods require training to prepare a model for the task at hand. Both approaches, hence, are not optimal for the requirements of efficiency and scalability in manufacturing. Chen et al. [2] state that for domain KGs “existing KG construction methods heavily rely on human intervention to attain qualified KGs, which severely hinders the practical applicability in real-world scenarios”. The authors exploit LLMs for KG construction – focussing on extraction from text – and achieve a result surpassing precision values of existing SOTA KG construction methods.

While multiple KG construction approaches acquire knowledge from unstructured data sources, Mapping Generation from structured sources such as relational databases differs, e.g., in the availability of a structure (schema). Utilizing this schema during knowledge extraction can prove helpful, as seen in e.g., Medeiros et al. [3] who introduce MIRROR, a system to generate R2RML mappings. One mapping type produces RDF triples homomorphic to the ones generated by a DM approach [3]. Another results in additional mappings containing implicit knowledge from database schemas [3]. Also, Hazber et al. [9] integrate the database schema R2RML mapping generation.

Next, we review first approaches on LLM-supported mapping generations. Hofer et al. explore LLM-supported RML mapping generation. The authors see promising results of commercial LLMs, such as Claude3 Opus (20240229) and GPT4 Turbo (01-25-preview) with zero-shot and on a subset from the Internet Movie Database. Our approach differs from these works in three aspects: (1) we not only explore RML, but further YARRRML syntax, (2) we investigate necessary ontology reduction, and (3) we evaluate the performance on real manufacturing data from a plant. There exist benchmark tasks [6] for LLM performance evaluation on KG engineering tasks, e.g., parsing and creating KGs serialized in Turtle syntax. Frey et al. [7] assess the evolution in Turtle skills of selected commercial LLMs on such tasks. While they show an improvement on the majority of tasks of the LLMs over their predecessors, the authors observe different responses from the models, especially on the output format, and conclude that stricter multi-shot tests are worth evaluating. They further see an open question on the evaluation on RML related tasks. We address both aspects for RML and YARRRML syntax. Thoroughly assessing the mapping quality is needed from a practical manufacturing perspective, yet research shows gaps in this area. Dimou et al. [4] introduce a quality assessment and a refinement workflow for RDF mappings. There is further a “framework to assess the quality of RDF mappings” [14] in YARRRML syntax, yet, the set of metrics is not validated against an existing ontology as in our use case. Some approaches on LLM-supported and context-enhanced mapping generations have been proposed and case-study-explored [11]. To the best of our knowledge, none of the approaches have been applied in a real manufacturing setting and none considers the YARRRML syntax. The previous works do not take an existing target ontology into consideration, which is a constraint in our setting. As a result, state-of-the-art solutions are insufficient to address our use case. We address these research gaps in our work.

In this paper, we introduce MYAM+R, an approach for LLM-supported and context-enhanced YARRRML and RML mapping generation, as a support for manufacturing knowledge experts. MYAM+R comprises of eight modules: four for input, as well as one for ontology reduction, mapping generation, evaluation and KG generation, each. The input modules focus on a data source sample, a related ontology, context enhancements, and prompt instructions for mapping generation. The mapping generation module can be configured with parameters for data size, ontology reduction method, and context enhancements. This module further orchestrates the processing steps, calls the ontology reduction module, and prepares the final prompt. Next, the generated mapping is evaluated against gold standard mappings which are created manually by experts. Finally, a KG generation module creates a KG based on the data and generated mapping. We conduct experiments with different configurations of MYAM+R across three real world data sets from Bosch. Naive ontology reduction and a mapping template as enhancement show promising as they achieve with column headers and first twenty data rows as input the best performance of F1=0.74 for YARRRML and F1=0.70 for RML mappings across all data sets.
We achieve our objective in supporting experts in manufacturing mapping generation with the obtained results and feedback. Furthermore, our results present a baseline for future experiments in the manufacturing domain. We publish the source code¹⁰¹⁰10https://github.com/boschresearch/myamr_tgdk to support further research in this direction.
In future work, next to addressing the limitations discussed in Section 4, we enhance MYAM+R in several aspects. We will improve the implementation with ontology learning from data and matching with the given ontology. With this, we will evaluate MYAM+R on domain and upper ontologies. Further, it is promising to extend the input configuration with a representative sample from the data input. We will explore mapping quality improvements with multiple prompts and sequential chain approaches. With the intention to increase the level of support for mapping generation in manufacturing, we explore different research avenues, e.g., the inclusion of other LLMs as well as fine-tuned small language models for a given domain.