Document Open Access Logo

GraphRAG on Technical Documents - Impact of Knowledge Graph Schema

Authors Henri Scaffidi , Melinda Hodkiewicz , Caitlin Woods , Nicole Roocke

PDF

File

Thumbnail PDF
PDF
TGDK.3.2.3.pdf
  • Filesize: 1.72 MB
  • 24 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Information systems → Specialized information retrieval
Keywords
  • RAG
  • minerals
  • local search
  • global search
  • entity extraction
  • competency questions

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Retrieval Augmented Generation (RAG) is seeing rapid adoption in industry to enable employees to query information captured in proprietary data for their organisation. In this work, we test the impact of domain-relevant knowledge graph schemas on the results of Microsoft’s GraphRAG pipeline. Our approach aims to address the poor quality of GraphRAG responses on technical reports rich in domain-specific terms. The use case involves technical reports about geology, chemistry and mineral processing published by the Minerals Research Institute of Western Australia (MRIWA). Four schemas are considered: a simple five-class minerals domain expert-developed schema, an expanded minerals domain schema, the Microsoft GraphRAG auto-generated schema, and a schema-less GraphRAG. These are compared to a conventional baseline RAG. Performance is evaluated using a scoring approach that accounts for the mix of correct, incorrect, additional, and missing content in RAG responses. The results show that the simple five-class minerals domain schema extracts approximately 10% more entities from the MRIWA reports than the other schema options. Additionally, both the five-class and the expanded eight-class minerals domain schemas produce the most factually correct answers and the fewest hallucinations. We attribute this to the minerals-specific schemas extracting more relevant, domain-specific information during the Indexing stage. As a result, the Query stage’s context window includes more high-value content. This contributes to the observed improvement in answer quality compared to the other pipelines. In contrast, pipelines with fewer domain-related entities in the KG retrieve less valuable information, leaving more room for irrelevant content in the context window. Baseline RAG responses were typically shorter, less complete, and contained more hallucinations compared to our GraphRAG pipelines. We provide a complete set of resources at https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain/tree/main. These resources include links to the MRIWA reports, a set of questions (from simple to challenging) along with domain-expert curated answers, schemas, and evaluations of the pipelines.

Cite As Get BibTex

Henri Scaffidi, Melinda Hodkiewicz, Caitlin Woods, and Nicole Roocke. GraphRAG on Technical Documents - Impact of Knowledge Graph Schema. In Transactions on Graph Data and Knowledge (TGDK), Volume 3, Issue 2, pp. 3:1-3:24, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/TGDK.3.2.3

Author Details

Henri Scaffidi
  • School of Computer Science and Software Engineering, The University of Western Australia, Australia
Melinda Hodkiewicz
  • School of Mechanical Engineering, The University of Western Australia, Australia
Caitlin Woods
  • School of Computer Science and Software Engineering, The University of Western Australia, Australia
Nicole Roocke
  • Minerals Research Institute of Western Australia, Australia

Funding

  • Roocke, Nicole: MRIWA Grant No. M10530. DOI: https://doi.org/10.71342/574784944778

Supplementary Materials

References

  1. Bradley P Allen, Lise Stork, and Paul Groth. Knowledge engineering using large language models. arXiv preprint arXiv:2310.00637, 2023. URL: https://doi.org/10.48550/arXiv.2310.00637.
  2. Tyler Bikaun, Michael Stewart, and Wei Liu. Quickgraph: A rapid annotation tool for knowledge graph extraction from technical text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, 2022. URL: https://doi.org/10.18653/v1/2022.acl-demo.27.
  3. Kurt Bollacker, Robert Cook, and Patrick Tufts. Freebase: A shared database of structured general human knowledge. In AAAI, volume 7, pages 1962-1963, 2007. URL: http://www.aaai.org/Library/AAAI/2007/aaai07-355.php.
  4. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1-45, 2024. URL: https://doi.org/10.1145/3641289.
  5. Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for RAG systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 719-729, 2024. URL: https://doi.org/10.1145/3626772.3657834.
  6. Armita Davarpanah, Hassan A Babaie, and W Crawford Elliott. Knowledge-based query system for the critical minerals. Applied Computing and Geosciences, 22:100167, 2024. URL: https://doi.org/10.1016/j.acags.2024.100167.
  7. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A Graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024. URL: https://doi.org/10.48550/arXiv.2404.16130.
  8. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of Retrieval Augmented Generation. arXiv preprint arXiv:2309.15217, 2023. URL: https://doi.org/10.48550/arXiv.2309.15217.
  9. Mathieu Fenniak and PyPDF2 Contributors. PyPDF2: A pure‐Python PDF library, version 3.x. GitHub, 2022. Accessed: 17‑Jul‑2025. URL: https://github.com/py-pdf/pypdf.
  10. E. Friis. Langchain. GitHub, 2024. Accessed: 09-May-2024. URL: https://github.com/langchain-ai.
  11. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-Augmented Generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023. URL: https://doi.org/10.48550/arXiv.2312.10997.
  12. Luan Fonseca Garcia, Mara Abel, Michel Perrin, and Renata dos Santos Alvarenga. The geocore ontology: a core ontology for general use in geology. Computers & Geosciences, 135:104387, 2020. URL: https://doi.org/10.1016/j.cageo.2019.104387.
  13. Tania Hidalgo, Laura Kuhar, Andreas Beinlich, and Andrew Putnis. Understanding fluid–rock interactions and lixiviant/oxidant behaviour for the in‑situ recovery of metals from deep ore bodies. MRIWA Report M0488, Minerals Research Institute of Western Australia, 2020. Report No. 488. Google Scholar
  14. International Organization for Standardization. ISO/IEC 21838-1:2021 Information technology — Top-level ontologies (TLO) - Part 1: Requirements. International standard, International Organization for Standardization, 2021. Google Scholar
  15. Xinke Jiang, Ruizhe Zhang, Yongxin Xu, Rihong Qiu, Yue Fang, Zhiyuan Wang, Jinyi Tang, Hongxin Ding, Xu Chu, Junfeng Zhao, et al. Think and retrieval: A hypothesis knowledge graph enhanced medical large language models. arXiv preprint arXiv:2312.15883, 2023. URL: https://doi.org/10.48550/arXiv.2312.15883.
  16. Minki Kang, Jin Myung Kwak, Jinheon Baek, and Sung Ju Hwang. Knowledge graph-augmented language models for knowledge-grounded dialogue generation. arXiv preprint arXiv:2305.18846, 2023. URL: https://doi.org/10.48550/arXiv.2305.18846.
  17. Jerry Liu. Llamaindex. GitHub, 2022. Accessed: 09-May-2024. URL: https://github.com/jerryjliu/llama_index.
  18. Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061, 2023. URL: https://doi.org/10.48550/arXiv.2310.01061.
  19. Microsoft. GraphRAG. https://microsoft.github.io/graphrag/. Accessed: 2024-09-30.
  20. Microsoft. GraphRAG. https://github.com/microsoft/graphrag/tree/v0.1.1, 2024. Accessed: 2024-09-30.
  21. Microsoft. GraphRAG. https://github.com/microsoft/graphrag/tree/v2.3.0, 2025. Accessed: 2025-07-25.
  22. Mineral Research Institute of Western Australia. Project Portfolio. https://www.mriwa.wa.gov.au/research-projects/project-portfolio/, 2024. [Online; accessed 9-May-2024].
  23. Pin Ni, Ramin Okhrati, Steven Guan, and Victor Chang. Knowledge graph and deep learning-based text-to-graphql model for intelligent medical consultation chatbot. Information Systems Frontiers, 26(1):137-156, 2024. URL: https://doi.org/10.1007/s10796-022-10295-0.
  24. Daniela Oliveira, Ratnesh Sahay, and Mathieu d'Aquin. Leveraging ontologies for knowledge graph schemas. In KGB@ ESWC, pages 24-36, 2019. URL: https://ceur-ws.org/Vol-2489/paper3.pdf.
  25. Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, 36:3580-3599, 2024. URL: https://doi.org/10.1109/tkde.2024.3352100.
  26. Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph Retrieval-Augmented Generation: A survey. arXiv preprint arXiv:2408.08921, 2024. URL: https://doi.org/10.48550/arXiv.2408.08921.
  27. Tyler Thomas Procko and Omar Ochoa. Graph Retrieval-Augmented Generation for large language models: A survey. In 2024 Conference on AI, Science, Engineering, and Technology (AIxSET), pages 166-169. IEEE, 2024. Google Scholar
  28. David Robinson. Improving solvent extraction technology. MRIWA Report M0401, Minerals Research Institute of Western Australia, 2010. Report No. 289. Google Scholar
  29. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An automated evaluation framework for Retrieval-Augmented Generation systems. arXiv preprint arXiv:2311.09476, 2023. URL: https://doi.org/10.48550/arXiv.2311.09476.
  30. Henri Scaffidi, Melinda Hodkiewicz, Caitlin Woods, and Nicole Roocke. nlp-tlp/GraphRAG-on-Minerals-Domain. InteractiveResource (visited on 2025-10-07). URL: https://github.com/nlp-tlp/GraphRAG-on-Minerals-Domain
    full metadata available at: https://doi.org/10.4230/artifacts.24886
  31. Juan Sequeda, Dean Allemang, and Bryon Jacob. A benchmark to understand the role of knowledge graphs on large language model’s accuracy for question answering on enterprise SQL databases. In Proceedings of the 7th Joint Workshop on Graph Data Management Experiences & Systems (GRADES) and Network Data Analytics (NDA), pages 1-12, 2024. URL: https://doi.org/10.1145/3661304.3661901.
  32. Tanya Serry, Tonya Stebbins, Andrew Martchenko, Natalie Araujo, and Brigid McCarthy. Improving access to COVID-19 information by ensuring the readability of government websites. Health Promotion Journal of Australia, 34(2):595-602, 2023. URL: https://doi.org/10.1002/hpja.610.
  33. Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. arXiv preprint arXiv:2407.10457, 2024. URL: https://doi.org/10.48550/arXiv.2407.10457.
  34. Vincent A Traag, Ludo Waltman, and Nees Jan Van Eck. From Louvain to Leiden: guaranteeing well-connected communities. Scientific reports, 9(1):1-12, 2019. URL: https://doi.org/10.1038/s41598-019-41695-z.
  35. Denny Vrandečić. Wikidata: A new platform for collaborative data collection. In Proceedings of the 21st international conference on World Wide Web, pages 1063-1064, 2012. URL: https://doi.org/10.1145/2187980.2188242.
  36. Chengbin Wang, Xiaogang Ma, Jianguo Chen, and Jingwen Chen. Information extraction and knowledge graph construction from geoscience literature. Computers & Geosciences, 112:112-120, 2018. URL: https://doi.org/10.1016/j.cageo.2017.12.007.
  37. Xintao Wang, Qianwen Yang, Yongting Qiu, Jiaqing Liang, Qianyu He, Zhouhong Gu, Yanghua Xiao, and Wei Wang. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv:2308.11761, 2023. URL: https://doi.org/10.48550/arXiv.2308.11761.
  38. M. Wells, M. Aylmore, and B. McInnes. The geology, mineralogy and geometallurgy of EV materials deposits in Western Australia. MRIWA Report M532, Geological Survey of Western Australia, 2022. Report No. 228. Google Scholar
  39. Zhentao Xu, Mark Jerome Cruz, Matthew Guevara, Tie Wang, Manasi Deshpande, Xiaofeng Wang, and Zheng Li. Retrieval-Augmented Generation with knowledge graphs for customer service question answering. arXiv preprint arXiv:2404.17723, 2024. URL: https://doi.org/10.1145/3626772.3661370.
  40. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, et al. CRAG-comprehensive RAG benchmark. Advances in Neural Information Processing Systems, 37:10470-10490, 2024. URL: https://doi.org/10.48550/arXiv.2406.04744.
  41. Huaxuan Zhao, Yueling Pan, and Feng Yang. Research on information extraction of technical documents and construction of domain knowledge graph. IEEE Access, 8:168087-168098, 2020. URL: https://doi.org/10.1109/access.2020.3024070.
  42. Lingfeng Zhong, Jia Wu, Qian Li, Hao Peng, and Xindong Wu. A comprehensive survey on automatic knowledge graph construction. ACM Computing Surveys, 56(4):1-62, 2023. URL: https://doi.org/10.1145/3618295.
  43. Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, et al. Rageval: Scenario specific RAG evaluation dataset generation framework. arXiv preprint arXiv:2408.01262, 2024. URL: https://doi.org/10.48550/arXiv.2408.01262.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact