GraphRAG on Technical Documents – Impact of Knowledge Graph Schema

The RAG paradigm is still evolving and can be broadly classified into three processes: baseline RAG, advanced RAG and modular RAG [25]. Baseline RAG has two steps: retrieval and generation. In the retrieval step, text from the document is segmented into chunks, then encoded using an embedding model and stored in a vector database. The RAG system uses the same encoding model for the user query and computes a similarity score between the vectors for the query and the indexed chunk. The top-k chunks with the highest similarity score are used to generate a response to the prompt. Baseline RAG, or “Naïve” RAG, often struggles to identify appropriate chunks, especially when the query requires large amounts of information (exceeding the top-k restriction) to be retrieved and then aggregated across documents, and when the query requires the system to synthesise insights not explicitly stated in reports [11].

Advanced RAG uses pre-retrieval and post-retrieval strategies to improve indexing and streamline the retrieval process. For example, there is growing interest in utilising additional content, specifically structured data such as knowledge graphs, to enhance the retrieval stage in RAG. This is called GraphRAG [26, 27]. Specifically, altering the retrieval source structure, from unstructured text to graph-based knowledge, for example, can improve the quality of retrieved content, which helps to reduce LLM hallucinations [16] and improve summarisation performance [7]. A modular RAG framework introduces new modules into the pipeline, allowing for substitution or reconfiguration of specific steps, making the process more adaptable to specific use-cases beyond traditional sequential retrieval and generation [11].

A KG is a collection of triples, where each triple represents a relation (edge) between two entities (nodes): relation(Entity1, Entity2). The KG schema defines the types of entities and relations permissible in the KG. GraphRAG relies on a KG to retrieve relevant information. However, manually curating a KG is labor-intensive and time-consuming, often relying on crowd-sourced efforts like Freebase [3] and Wikidata [35]. As a result, automated KG construction methods are crucial for efficiently extracting structured knowledge from unstructured or semi-structured data, significantly reducing the manual burden [42]. The challenge lies in designing a KG schema that balances domain specificity with adaptability, ensuring the KG can incorporate new concepts, evolving terminology, and varied technical document structures.

Broadly, there are two approaches to KG schema development: top-down and bottom-up. The top-down approach develops a schema based on domain knowledge and the use case, sometimes informed by existing ontologies if available [24]. Bottom-up approaches identify concepts of interest using entity discovery processes to discover strings referring to semantic entities and then classify them into types, which collectively define the schema [36, 41, 42]. In practice, a combination of both approaches is usually deployed.

Within the minerals domain are several technical disciplines, each with its own processes, norms, and data models. These disciplines contribute at different stages along the mineral processing value chain – find and define the deposit (geology, geochemistry, geostatistics), consider options for processing (geometallurgy), identify risks and quantify the value (mining finance), mine planning and execution (mining engineering), mineral processing (metallurgists) to produce the saleable product (marketing), environmental management and closure (environmental engineers). These disciplines draw on the sciences – including geology, chemistry, physics, and mathematics – as well as all branches of engineering and business. While there has been work on developing schemas and ontologies in subdomains such as geology [6, 12], there is no publicly available KG schema or ontology that spans the entire minerals domain value chain. An ontology is a formal and semantically rich specification used when reasoning and interoperability are required.

As the literature on RAG has grown, more studies have sought to integrate KGs into the pipeline, particularly as the field acknowledges the limitations of vector search over unstructured text [11]. Broadly, these efforts can be categorised into two approaches: using LLM-generated KG queries [10, 17, 18, 37, 39], and machine learning-based KG querying [15, 16, 23].

Microsoft’s GraphRAG [7], introduced in 2024, uses LLMs to generate the KG from text. Graph machine learning is utilised to build a bottom-up hierarchical cluster of the KG using an LLM to pre-summarise each cluster and produce a set of community reports. The KG and community reports are retrieved via two search options: local search and global search. Local search identifies semantically related entities within the KG, based on the prompt, and retrieves closely connected information. This search method is suited to answer targeted queries about specific entities [19]. Global search retrieves all community reports at a specific level of the hierarchical cluster and conducts Map-Reduce to synthesise a response. This search method is designed to answer high-level aggregation queries about the dataset.

Given the novelty of GraphRAG (released July 2, 2024), there is no research testing it in the minerals domain. As shown in Table 1, the published literature on Microsoft GraphRAG is limited. In our analysis, surveys were excluded, and only studies that evaluated Microsoft GraphRAG’s question–answering performance were included. Across the 13 identified studies, most used broad, general-domain public datasets such as UltraDomain or HotpotQA, while only a small proportion applied GraphRAG to narrow industrial and technical domain corpora. LLM-based judging was the predominant evaluation approach, with few examples of human assessment or automated retrieval metrics. Citation counts are generally low, and impact factors, where applicable, are modest. These trends highlight a gap in the literature, which our work addresses by evaluating GraphRAG’s performance in a real-world context within the minerals domain using human subject-matter expert evaluation.

Many organisations are seeking to use RAG to query their internal documents, and given their existing commitment to the Microsoft ecosystem, Microsoft’s GraphRAG has become a relevant product worth investigating.

Based on the gaps in the literature discussed in this section we propose the following research question: Given a set of minerals research reports that are accessed in a GraphRAG pipeline, how do different KG schemas of the report’s subject matter impact the quality of responses to local and global search queries?

The validation input for our analysis consists of a set of competency questions to assess the effectiveness of a minerals domain RAG pipeline. The questions were developed by MRIWA’s CEO and several members of the MRIWA staff. As representatives of the typical users of the RAG pipeline, they provided questions that reflect real-world information needs in the minerals research sector. The quantity of information required to answer each question and the complexity of each question varies. We have categorised these questions into five groups based on the CRAG [40] classification scheme: simple, simple with condition, set, aggregation, and post-processing-heavy (see Table 2 and Table 3). The CRAG paper defines eight question types, five of which apply to our question set. The three CRAG question types not covered by the CEO’s questions are comparison, multi-hop, and false-premise.

For each competency question the MRIWA team developed an ideal answer²²2https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/data/mriwa_cqa. These ideal answers are designed specifically for use with our evaluation approach outlined in Section 4.5.

Some of the queries test GraphRAG’s ability to find and return concepts not explicitly referenced in the query. For example, the question “Which MRIWA report is related to the East Kimberley region” requires the model to identify towns, mine sites, and geological locations in the East Kimberley region, not rely on the regular expression East Kimberley. Similarly, for “reports relating to leaching” we hope to find reports relating to leaching from a minerals processing perspective (the context of the question) and which mention any chemical process that extracts valuable minerals from ore by dissolving with a solvent.

A subset of MRIWA’s reports is sampled based on domain-expert opinion informed by natural language processing to count unique tokens, for instance. The selection process aims to ensure the data set covers multiple minerals domains and represents the complexity of the report set. A subset of 15 reports was selected, totalling approximately 2.7 million tokens – comparable in scale to the datasets used in the original Microsoft GraphRAG paper [7]. We believe this scale increases both the reliability of our results and the usefulness of the resources. The selection process is described in detail in the GitHub repository³³3https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/blob/main/supplementary_materials/mriwa_report_sample_selection.md. The selected PDF reports were converted to individual “.txt” files using the PyPDF2 [9] Python package for use in subsequent steps.

The development of a minerals-domain KG schema began with a review of the concepts in MRIWA report executive summaries. These executive summaries (300-1000 words) provide an overview of topics and domain concepts covered in the full report content. Initially, we explored annotation classes (Activity, Physical Object, Process, Property, and State) used in our previous industrial annotation work⁴⁴4https://github.com/nlp-tlp/maintie. This resulted in almost every item in the text having an annotation, and much of what was being annotated was irrelevant to addressing the questions being asked. We then explored discipline-specific (e.g., geology and mining) entities, but these were difficult to annotate consistently due to the spans. Following this experience, and with the assistance of MRIWA subject matter experts, we landed on more abstract concepts like Naturally Occurring Object to describe geological formations (present in many of the reports) and separated these from Processed Object, which are produced in some engineered process. We developed separate classes for objects, processes, location-specific concepts and for identifying organisations. We called this the Minerals Domain Schema (MDS). Early tests showed GraphRAG performed well with these aggregated concepts.

We then extended this schema to include some additional subclasses, splitting Processed Object into Processed Material and Manufactured Product (mirroring the mineral processing and manufactured output in mining), and sub-classing Process into Natural Process, Lab testing Process and Industrial Process. This was called the Expanded Minerals Domain Schema (EMDS). For a summary of these two schemas, see Table 4. We used our QuickGraph annotation tool⁵⁵5https://quickgraph.tech/ [2] throughout this iterative process. Figure 2 provides a detailed example of entity typing under the EMDS from our manual annotation sessions. If the MDS were applied instead, the same entities would be labelled as follows:

• $\mathrm{■}$

  Entities labelled as Natural_Process, Lab_Process, or Industrial_Process would all be labelled as Process.

• $\mathrm{■}$

  Entities labelled as Processed_Material or Manufactured_Product would both be labelled as Processed_Object.

In considering the entity types in the MDS and EMDS, we also considered the possibility that future work might wish to align the schema classes to the Basic Formal Ontology (BFO) [14] or another top level ontology. Thus, the entity types Naturally_Occurring_Object, Processed_Object and Manufactured_Product can be mapped to the BFO class bfo:MaterialEntity, Process to bfo:Process, Organisation to bfo:ObjectAggregate, and Site_Location_Boundary to bfo:ImmaterialEntity.

Whilst defining relationships between entities would render the schemas more expressive, we do not, as GraphRAG [7] supports only the pre-definition of entity types. Relations are generated dynamically by the LLM during Indexing (see Section 4.3.1). Results of GraphRAG entity extraction using the schemas, including quantities of entities extracted, are discussed in Section 5.1.1.

Microsoft’s GraphRAG approach [7] is designed to answer complex semantic queries and aggregation queries. Figure 3 shows the two GraphRAG phases: Indexing and Query.

The experiment utilises GraphRAG Version 0.1.1 [20] in its default configuration with GPT-4o-Mini (temperature: 0.0). We run GraphRAG Indexing in four separate pipelines, each using one of the following KG schemas as the entity_types parameter for entity and relation extraction.

1. 1.

   Minerals Domain Schema (MDS):

   See Table 4. The LLM is not told the descriptions of the schema’s entity types, only the names of the entity types. Pipeline configuration⁶⁶6https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/src/minerals_domain_schema.

2. 2.

   Expanded Minerals Domain Schema (EMDS):

   See Table 4. The LLM is not told the descriptions of the schema’s entity types, only the names of the entity types. Pipeline configuration⁷⁷7https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/src/expanded_minerals_domain_schema.

3. 3.

   Auto-Generated Schema (AGS):

   This schema is generated by an LLM through the GraphRAG Prompt Tuner. We utilise the default LLM prompts, not the auto-tuned LLM prompts, to ensure a fair comparison between the other schema pipelines. We only utilise the entity type names generated by the Prompt Tuner, as follows.

   {chemical process, mineral, geological survey, geochemistry, exploration
   technique, sample, formation, project, research study}

   Pipeline configuration⁸⁸8https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/src/auto_generated_schema.

4. 4.

   Schema-less (SL):

   This pipeline leaves the KG schema unspecified, and instead prompts the LLM to “identify all entities needed from the text in order to capture the information and ideas in the text”. Pipeline configuration⁹⁹9https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/src/schema_less.

Additionally, we included Baseline RAG for comparison purposes. For this, we utilise a basic search pipeline from GraphRAG Version 2.3.0 [21] in its default configuration with GPT-4o-Mini (temperature: 0.0, chunk size: 300 tokens, chunk overlap: 100 tokens). The pipeline configuration is available¹⁰¹⁰10https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/supplementary_materials/basic_search.

Our code is available at https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/src.

Each of the four pipelines, through both local and global search, as well as the basic search pipeline, are asked 15 competency questions defined by MRIWA (see Table 3). Results can be accessed at https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/results.

Evaluation of RAG systems focuses on two key aspects: retrieval quality, which refers to the usefulness of retrieved information for answering a query, and generation quality, which measures the ability to generate accurate and relevant responses from the retrieved information [11]. GraphRAG uses our minerals reports and summarises them into a KG. It is not plausible to define a correct set of nodes/summaries to retrieve for a certain query, given that the number of nodes in the generated KG exceeds 195,000 for each of our trials. This precludes the use of many traditional metrics of retrieval quality, including Hit Rate and Mean Reciprocal Rank, which focus on the frequency at which the system retrieves the “correct” piece of information. Studies that have existing SQL or Graph databases are able to define concrete queries or answers for which the LLM’s output can be compared, enabling the use of metrics including Average Overall Execution Accuracy [31]. This technique is also not possible with the size of the GraphRAG-generated database of our technical reports. Our evaluation of generation quality is constrained to the use of our MRIWA-defined ideal answers, which contain key information expected in a completely correct response, as a basis of comparison. Given the technical nature of MRIWA’s reports, however, this process of comparison requires subject matter expertise which further precludes using automated LLM-based evaluation approaches, like Retrieval Augmented Generation Assessment (RAGAS) [8] and Automated RAG Evaluation System (ARES) [29], which use non-expert general-purpose LLMs to conduct the evaluation.

For example, the answer to our simplest question “identify which MRIWA reports reference MERIWA or MRIWA" should be a simple list of report identifiers (MRIWA reports 432, 448, etc.). Instead, when using GraphRAG with the auto-generated schema, the response (shown below) is very verbose, containing a subset of the correct report numbers as well as information about the reports that was not requested and in some cases not completely correct.

“The following MRIWA reports reference either MERIWA or MRIWA:
1. **MRIWA REPORT M532**: This report focuses on the geology, mineralogy, and geometallurgy of electric vehicle (EV) materials deposits in Western Australia. It highlights the significance of lithium production and provides insights into local resources associated with these deposits. The report is a comprehensive study that contributes to understanding the characteristics and potential of EV materials in the region [Data: Reports (12493); Entities (290, 10929, 1445, 23142, +more)].
2. **MRIWA REPORT NO. 488**: This report details research findings on fluid-rock interactions and lixiviant/oxidant behavior for metal recovery from deep ore bodies. It is part of a broader initiative to understand the processes involved in mineral extraction and recovery [Data: Sources (9809, 9748, 9861); Entities (290, 10929, 1445, 23142, +more)].
3. **FINAL REPORT TO MRIWA**: This document encapsulates the findings and analyses from the By-products (LEAF) Project M451, detailing the project’s outcomes, methodologies, and implications regarding soil amendments and their effects on leachate. It serves as a vital resource for understanding the impacts of soil amendments on leachate and the broader environmental context [Data: Reports (11245); Entities (290, 10929, 1445, 23142, +more)].
These reports illustrate the collaborative efforts and research initiatives overseen by MRIWA, contributing to advancements in mineral research and environmental management in Western Australia. The references to MRIWA in these reports underscore its role in funding and overseeing significant research projects in the region."

Evaluation of long-form answers, like the above, in technical domains is in its infancy, and traditional benchmarking methods that rely on standard train/test datasets have shown limitations in assessing LLM-generated responses [1, 4]. The process still depends, in large part, on evaluation against domain-expert provided model answers.

To evaluate these RAG pipelines on the competency questions we used a human-in-the-loop approach, incorporating subject matter evaluation and reference to the ideal answers.

Our classification approach is informed by evaluation measures for 1) Quality (Context Relevance, Answer Faithfulness, Answer Relevance) and 2) Required Abilities (Noise Robustness, Negative Rejection, Information Integration, and Counterfactual Robustness) [11]. Rather than provide an assessment against each of these seven dimensions, which was very time-consuming and difficult to agree on, we created a five-level classification approach shown in Table 6. We found that the wording enabled evaluators to provide a yes/no answer for each level. While based on the aforementioned dimensions, our classification scheme is coherent with the metrics proposed in RAGEval [43]: Completeness (B1 and B2), Irrelevance (B3 and B4), and Hallucination (B5).

Two independent raters classified all 135 responses to the competency questions using the classification scheme. Each question’s ideal answer outlined the specific information required for a response to be considered fully correct. Prior to rating, the raters reviewed each ideal answer and agreed on how the classification scheme would apply to that particular question, ensuring consistent interpretation. For example, the question “Identify which MRIWA reports reference MERIWA or MRIWA” has the following ideal answer and agreed upon evaluation scheme:

Final-Report_448_MRIWA_M0448.txt: MRIWA
Final-Report_278_MERIWA_M0384.txt: MERIWA
Final-Report_289_MERIWA_M0401.txt: MERIWA
Final-Report_532-MRIWA_M0532.txt: MRIWA
Final-Report_479_MRIWA_M0479.txt: MRIWA
Final-Report_459-489_MRIWA_M0459-M0484.txt: MRIWA
Final-Report_458_MRIWA_M0458.txt: MRIWA
Final-Report_476_MRIWA_M0476.txt: MRIWA
Final-Report_432_MRIWA_M0432.txt: MRIWA
Final-Report_470_MRIWA_M0470.txt: MRIWA
Final-Report_295_MERIWA_M0416.txt: MERIWA
Final-Report_494_MRIWA_M0494.txt: MRIWA, MERIWA
Final-Report_488_MRIWA_M0488.txt: MRIWA
Final-Report_451_MRIWA_M0451.txt: MRIWA
Final-Report_505_MRIWA_M0505.txt: MRIWA

B1 – All above reports noted as referencing MRIWA/MERIWA.
B2 – Some of above reports noted as referencing MRIWA/MERIWA.
B3 – Discussions about MRIWA/MERIWA in MRIWA’s reports generally, but not specifying or referring to the specific MRIWA report/s that mention MRIWA/MERIWA.
B4 – Discussions not about or unrelated to MRIWA/MERIWA in MRIWA’s reports (e.g., MRIWA appearing in non-MRIWA reports).
B5 – False statement.

Multiple performance classifications (B1 – B5) may be attributed to a single response. For example, a response may contain “all ideal answer information” (B1) from our MRIWA-defined ideal answer but also include “some factually correct information that is irrelevant to the question” (B4).

The evaluation results (averaged across the raters) using the five-level classification system for each GraphRAG pipeline (with four KG schema options, and local/global search) and the Baseline RAG pipeline are shown in Table 7. All pipeline responses are available¹¹¹¹11https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main/supplementary_materials/performance_analysis. As described in Section 4.5, the GraphRAG responses contain multiple paragraphs, relevant and irrelevant answers and facts, and in some cases reference the retrieved information. The average length of responses is 302.40 words (396.97 tokens).

We make the following observations about the schema pipeline data in Table 7 before presenting in-depth discussions on the performance of specific KG schemas in the following subsections. In discussing the performance classifications, we consider a good performance as having a high total for B1 and B2 (relevance) and a low total for B5 (hallucinations). B3 and B4 (robustness) are informative in that a high total indicates that additional peripheral information is being returned that may or may not be relevant to the question.

Relevance (B1 and B2):

• $\mathrm{■}$

  All schemas exhibit similar levels of relevance (B1 and B2).

• $\mathrm{■}$

  57% of responses exhibit B2 (contain some ideal answer information) indicating majority of responses are partially correct.

• $\mathrm{■}$

  Only approximately 7 of the 120 responses received a B1 score (all ideal answer information present in the response).

• $\mathrm{■}$

  Local search slightly outperforms global search on relevance (B1 and B2).

Robustness (B3 and B4):

• $\mathrm{■}$

  Global search pipelines produced more irrelevant information (B4) than local search pipelines.

• $\mathrm{■}$

  The MDS pipeline with local search produced fewer examples of peripheral information (B3 and B4) than MDS using global search.

• $\mathrm{■}$

  The MDS pipeline produced the second highest amount of peripheral information (B3 and B4) and produced the most relevant peripheral information (B3).

Hallucination (B5):

• $\mathrm{■}$

  The MDS pipeline produced hallucinations (B5) at a lower rate (though comparable with EMDS) than the other pipelines, only hallucinating on the following four competency questions: Two Simple queries “Which MRIWA reports has Commonwealth Scientific Industrial Research Organisation been involved with in any capacity (including being listed in references)?”, “Which MRIWA report is related to the East Kimberley region?”, one Simple with Condition query “Which MRIWA reports has Commonwealth Scientific Industrial Research Organisation been involved with as a sponsor?”, and one Aggregation query “What is the average number of references in each MRIWA report?”

• $\mathrm{■}$

  All local search variants hallucinated on the Aggregation query “What is the average number of references in each MRIWA report?", while none of the global search pipelines hallucinated.

• $\mathrm{■}$

  The highest number of hallucinations is from the AGS pipeline using global search, hallucinating on four out of the 15 competency questions.

General Observations:

• $\mathrm{■}$

  The MDS and EMDS pipelines using local search had the most desirable set of scores.

• $\mathrm{■}$

  Global search results of all pipelines are not clear to rank in terms of performance.

In this section we explore the impact of the type of question on the performance of the MDS GraphRAG pipeline by looking at some specific examples in more detail. We chose the MDS pipeline as it has strong performance compared to the other pipelines, as discussed in the preceding sections.

To investigate the depth of answers, we do a phrase-by-phrase analysis of the responses to the question, “Which MRIWA report is related to the East Kimberley region?”. We classify phrases into five categories, as is displayed in Figure 6. These figures aim to give the reader a visual sense of the complexity of evaluating RAG responses. We want to see answers with cool colours (green and blue) and without hot colours (red and pink).

In general, pipelines using a MRIWA-specific schema produce responses with a larger quantity of correct and relevant statements, compared to the auto-generated schema and schema-less pipelines. This finding is consistent with the results of our performance analysis.

The Local Search responses tend to be more concise and contain more correct and relevant statements (blue and green) than global search responses. This result is consistent with the fact that low-level targeted queries, such as “Which MRIWA report is related to the East Kimberley region?” are answered better with Local Search.

Default GraphRAG usually provides an unasked-for final paragraph starting with the words “overall”, “in conclusion”, “in summary”, or “this report highlights”. These paragraphs usually contain no new information. This behaviour may be altered by changing the default GraphRAG LLM prompts and parameters. For example, the response_type parameter of the LocalSearch and GlobalSearch classes allow a description of the response format to be specified – the parameter defaults to “Multiple Paragraphs”.

None of the pipelines identify MRIWA Report 448 as being related to the East Kimberley region. While the region is not a primary focus of the report, mentions of the Kimberley appear in several data tables and references. However, these mentions do not occur near the term “Report 448” (or equivalent). Since the report is initially split into 300-token chunks, “Report 448” and “Kimberley” do not appear in the same chunk. As a result, the LLM fails to associate Report 448 with the Kimberley, leading to its omission in responses. Capturing document structure and metadata may help trace the source report of specific entities.

Among the pipelines that correctly identify MRIWA Report 459-484 as primarily focused on the East Kimberley, all fail to capture key details from the report’s abstract when generating a summary. This result suggests GraphRAG’s summarisation process during Indexing may lose important depth typically found in a technical report abstract.

GPT-4o-Mini is a non-deterministic LLM. Therefore, repetitions of the experiment are likely to yield different results [33]. The impact of non-determinism may be prevalent during entity and relation extraction. Given the same KG schema and set of text chunks, the entities and relations identified may not be identical between trials, resulting in variation of the final KG, impacting GraphRAG responses. As Edge et al. [7] note, this issue may be solved by conducting multiple gleanings, where the process of entity extraction is repeated to detect additional entities missed by the LLM.

A risk of using OpenAI’s GPT is its closed-source nature, which can lead to vendor lock-in and raise privacy and confidentiality concerns. Additionally, changes to the underlying model may impact repeatability. While using an open-source LLM like Llama can provide more control, it does not eliminate the possibility of updates affecting results and may introduce performance differences.

Finally, our experiments utilise a subset of 15 MRIWA reports and do not include tables or figures from the reports. These may provide valuable information for the construction of a more comprehensive KG. However, all the MRIWA reports are publicly available, and for the 15 reports we have used, we provide the questions, our ideal answers, schemas, and details on how we performed each evaluation. This will allow others to replicate and extend our work.