Document Open Access Logo

CognixShield: PoV-Guided Vulnerable API Usage Detection in Large Codebases via LLMs

Authors Quanzhi Fu , Wang Lingxiang , Wenjia Song , Gelei Deng , Yi Liu , Dan Williams , Ying Zhang

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.1.pdf
  • Filesize: 1.19 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Security and privacy → Software security engineering
Keywords
  • Vulnerable API usage detection
  • program analysis
  • LLMs
  • agentic RAG

Metrics

Abstract

Background. The integration of open-source libraries in Java development introduces severe security risks through vulnerable APIs. Existing program analysis and deep learning tools face challenges in capturing inter-procedural vulnerability semantics at scale. While LLMs show promise for semantic reasoning, they cannot handle large codebases due to context limits, and they lack the vulnerability-specific understanding needed to determine exploitability.

Aim. This work aims to overcome these limitations and enable LLM-based detection of vulnerable API usage in large-scale Java applications.

Method. We present CognixShield, an LLM-powered framework for detecting vulnerable API usage through three core components. First, semantic-preserving AST-based fragmentation partitions large codebases while maintaining syntactic completeness within LLM context windows. Second, vulnerability-aware multi-agent RAG traces relevant program context across these fragments, iteratively assembling security-critical context spanning functions and files. Third, PoV-guided semantic reasoning uses Proof-of-Vulnerability tests that encode precise triggering conditions and exploitation mechanics to determine vulnerability.

Results. CognixShield achieves 84% precision, 95% recall, 84% accuracy, and an 89% F1-score on 57 real-world Java applications, outperforming state-of-the-art tools.

Conclusions. Our results show that vulnerability detection requires specialized architectural innovations beyond generic LLM applications.

Cite As Get BibTex

Quanzhi Fu, Wang Lingxiang, Wenjia Song, Gelei Deng, Yi Liu, Dan Williams, and Ying Zhang. CognixShield: PoV-Guided Vulnerable API Usage Detection in Large Codebases via LLMs. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 1:1-1:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.1

Author Details

Quanzhi Fu
  • Virginia Tech, Blacksburg, VA, USA
Wang Lingxiang
  • Independent Researcher, Sunnyvale, CA, USA
Wenjia Song
  • Virginia Tech, Blacksburg, VA, USA
Gelei Deng
  • Nanyang Technological University, Singapore
Yi Liu
  • Griffith University, Gold Coast, Australia
Dan Williams
  • Virginia Tech, Blacksburg, VA, USA
Ying Zhang
  • Wake Forest University, Winston Salem, NC, USA

Acknowledgements

This work has been partially supported by the Wake Forest Pilot Research Grant. We thank the anonymous reviewers for their valuable comments on an earlier version of this paper.

Supplementary Materials

References

  1. OWASP Dependency-Check. https://owasp.org/www-project-dependency-check/, 2020.
  2. VAScanner. https://github.com/VAScanner/VAScanner, 2024. Accessed: 2024-01-22.
  3. A03:2025 Software Supply Chain Failures. https://owasp.org/Top10/2025/A03_2025-Software_Supply_Chain_Failures/, 2026.
  4. Dependency-check vulnerabilities | Snyk. https://security.snyk.io/package/npm/dependency-check, 2026.
  5. Meta AI. Meta llama 3.1 announcement. https://ai.meta.com/blog/meta-llama-3-1/, 2024. Accessed: 2024-10-30.
  6. Voyage AI. Voyage code 3. https://blog.voyageai.com/2024/12/04/voyage-code-3/. Published: 2024-12-04, Accessed: 2025-01-20.
  7. Alibaba. PasswordEncoderUtil.java/UserController.java - Apache Kylin, 2025. Accessed: 2025-01-19. URL: https://github.com/alibaba/nacos/tree/a19d3fd.
  8. Anthropic. Claude sonnet. https://www.anthropic.com/claude/sonnet, 2024. Accessed: 2024-10-30.
  9. Apache Software Foundation. KylinUserService.java - Apache Kylin, 2025. Accessed: 2025-01-19. URL: https://github.com/apache/kylin/tree/443c252.
  10. Apache Software Foundation. UserController.java - Apache Kylin, 2025. Accessed: 2025-01-19. URL: https://github.com/apache/kylin/tree/443c252.
  11. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Self-reflective retrieval augmented generation. arXiv preprint, 2023. URL: https://arxiv.org/abs/2310.11511.
  12. Max Brunsfeld and contributors. Tree-sitter java. https://github.com/tree-sitter/tree-sitter-java, 2024. Accessed: 2024-10-30.
  13. Sicong Cao, Xiaobing Sun, Xiaoxue Wu, David Lo, Lili Bo, Bin Li, and Wei Liu. Coca: Improving and explaining graph neural network-based vulnerability detection systems. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1-13, 2024. URL: https://doi.org/10.1145/3597503.3639168.
  14. FindBugs Project Contributors. Bugtreemodel.java. https://github.com/tonydamage/findbugs/tree/fd7ec8b, 2023. Accessed: 2024-10-30.
  15. Yangruibo Ding, Jinjun Peng, Marcus J. Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. Semcoder: Training code language models with comprehensive semantics reasoning. In Advances in Neural Information Processing Systems, volume 37, 2024. Google Scholar
  16. Xueying Du, Geng Zheng, Kaixin Wang, Yi Zou, Yujia Wang, Wentai Deng, Jiayi Feng, Mingwei Liu, Bihuan Chen, Xin Peng, Tao Ma, and Yiling Lou. Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag. ACM Transactions on Software Engineering and Methodology, 2025. Google Scholar
  17. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural languages. arXiv preprint, 2020. URL: https://arxiv.org/abs/2002.08155.
  18. Felix Fischer, Konstantin Böttinger, Huang Xiao, Christian Stransky, Yasemin Acar, Michael Backes, and Sascha Fahl. Stack overflow considered harmful? the impact of copy&paste on android application security. In 2017 IEEE Symposium on Security and Privacy (SP), pages 121-136. IEEE, 2017. URL: https://doi.org/10.1109/SP.2017.31.
  19. Michael Fu and Chakkrit Tantithamthavorn. Linevul: A transformer-based line-level vulnerability prediction. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 608-620, 2022. URL: https://doi.org/10.1145/3524842.3528452.
  20. GitHub. Codeql, 2019. Accessed: 2025-08-05. URL: https://codeql.github.com/.
  21. Google. Gemini ai. https://gemini.google.com/. Accessed: 2025-01-20.
  22. Kai Huang, Xiangxin Meng, Jian Zhang, Yang Liu, Wenjie Wang, Shuhao Li, and Yuqing Zhang. An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair . In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1162-1174, September 2023. URL: https://doi.org/10.1109/ASE56229.2023.00181.
  23. Hong Jin Kang, Truong Giang Nguyen, Bach Le, Corina S Păsăreanu, and David Lo. Test mimicry to assess the exploitability of library vulnerabilities. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 276-288, 2022. Google Scholar
  24. Bastian Leicht. Badlion source code - uriutils.java, 2024. Accessed: 2024-10-30. URL: https://github.com/bastianleicht/badlion-src/tree/93a099e.
  25. Ahmed Lekssays, Hamza Mouhcine, Khang Tran, Ting Yu, and Issa Khalil. LLMxCPG: Context-aware vulnerability detection through code property graph-guided large language models. In 34th USENIX Security Symposium (USENIX Security 25), 2025. Google Scholar
  26. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459-9474, 2020. Google Scholar
  27. Qiang Li, Jinke Song, Dawei Tan, Haining Wang, and Jiqiang Liu. Pdgraph: A large-scale empirical study on project dependency of security vulnerabilities. In 2021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pages 161-173, 2021. URL: https://doi.org/10.1109/DSN48987.2021.00031.
  28. Yi Li, Shaohua Wang, and Tien N. Nguyen. Vulnerability detection with fine-grained interpretations. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2021, pages 292-303, 2021. URL: https://doi.org/10.1145/3468264.3468597.
  29. Zhen Li, Deqing Zou, Shouhuai Xu, Hai Jin, Yawei Zhu, and Zhaoxuan Chen. SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities . IEEE Transactions on Dependable and Secure Computing, 19(04):2244-2258, July 2022. URL: https://doi.org/10.1109/TDSC.2021.3051525.
  30. Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. Vuldeepecker: A deep learning-based system for vulnerability detection. In Network and Distributed Systems Security (NDSS) Symposium, 2018. Google Scholar
  31. Jinghua Liu, Yi Yang, Kai Chen, and Miaoqian Lin. Generating api parameter security rules with llm for api misuse detection. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2409.09288.
  32. Guilong Lu, Xiaolin Ju, Xiang Chen, Wenlong Pei, and Zhilong Cai. Grace: Empowering llm-based software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software, 212:112031, 2024. URL: https://doi.org/10.1016/J.JSS.2024.112031.
  33. National Vulnerability Database (NVD). Cve-2019-10094 - apache tika xml external entity (xxe) vulnerability, 2019. Accessed: 2025-03-14. URL: https://nvd.nist.gov/vuln/detail/CVE-2019-10094.
  34. OpenAI. Embedding models: Openai documentation. https://platform.openai.com/docs/guides/embeddings/embedding-models, 2024. Accessed: 2024-10-30.
  35. OpenAI. Model overview. https://platform.openai.com/docs/models, 2024. Accessed: 2024-10-30.
  36. OpenAI. OpenAI Codex CLI. https://github.com/openai/codex, 2025. Accessed: 2025.
  37. OWASP Foundation. OWASP Dependency-Check. https://owasp.org/www-project-dependency-check/, 2024. Accessed: 2024-01-22.
  38. Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. Examining Zero-Shot Vulnerability Repair with Large Language Models . In IEEE Symposium on Security and Privacy (SP), pages 2339-2356, May 2023. URL: https://doi.org/10.1109/SP46215.2023.10179324.
  39. JULIE PETERSON. The Scariest Things About SCA. https://devops.com/the-scariest-things-about-sca/, 2022.
  40. Serena Elisa Ponta, Henrik Plate, and Antonino Sabetta. Detection, assessment and mitigation of vulnerabilities in open source dependencies. Empirical Software Engineering, 25(5):3175-3215, 2020. URL: https://doi.org/10.1007/S10664-020-09830-X.
  41. Sazzadur Rahaman, Ya Xiao, Sharmin Afrose, Fahad Shaon, Ke Tian, Miles Frantz, Murat Kantarcioglu, and Danfeng (Daphne) Yao. Cryptoguard: High precision detection of cryptographic vulnerabilities in massive-sized java projects. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS '19, pages 2455-2472, 2019. URL: https://doi.org/10.1145/3319535.3345659.
  42. Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. Agentic retrieval‑augmented generation: A survey on agentic rag. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2501.09136.
  43. Larry Singleton, Rui Zhao, Myoungkyu Song, and Harvey Siy. Cryptotutor: Teaching secure coding practices through misuse pattern detection. In Proceedings of the 21st Annual Conference on Information Technology Education, pages 403-408, 2020. URL: https://doi.org/10.1145/3368308.3415419.
  44. spring-projects/spring-security. Fix issue (commit d1909ec). https://github.com/spring-projects/spring-security/commit/d1909ec, 2025. Accessed: 2025-07-29.
  45. Benjamin Steenhoek, Hongyang Gao, and Wei Le. Dataflow analysis-inspired deep learning for efficient vulnerability detection. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1-13, 2024. URL: https://doi.org/10.1145/3597503.3623345.
  46. Fute Sun, Lei Zhang, Zhiyu Wu, Xinyu Chen, Haitao Yu, Tianyang Han, and Min Yang. Fine-grained detection of java cross-library vulnerability propagation by extracting semantic constraints from security patches. IEEE Transactions on Dependable and Secure Computing, 23(4):8334-8348, 2026. URL: https://doi.org/10.1109/TDSC.2026.3682678.
  47. Tian Tan and Yue Li. Tai-e: A developer-friendly static analysis framework for java by harnessing the good designs of classics. In René Just and Gordon Fraser, editors, Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, Seattle, WA, USA, July 17-21, 2023, pages 1093-1105. ACM, 2023. URL: https://doi.org/10.1145/3597926.3598120.
  48. Qdrant Team. Qdrant: Vector database for the next generation of ai applications. https://qdrant.tech/, 2024. Accessed: 2024-10-30.
  49. Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks . In 2024 IEEE Symposium on Security and Privacy (SP), pages 862-880, May 2024. URL: https://doi.org/10.1109/SP54263.2024.00210.
  50. Wlainer. Basicnamefinder.java - ia recruiter, 2025. Accessed: 2025-03-14. URL: https://github.com/wlainer/ia-recruiter.
  51. Yulun Wu, Zeliang Yu, Ming Wen, Qiang Li, Deqing Zou, and Hai Jin. Understanding the threats of upstream vulnerabilities to downstream projects in the maven ecosystem. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1046-1058. IEEE, 2023. URL: https://doi.org/10.1109/ICSE48619.2023.00095.
  52. Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated Program Repair in the Era of Large Pre-trained Language Models . In IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1482-1494, May 2023. URL: https://doi.org/10.1109/ICSE48619.2023.00129.
  53. Kazuki Yamada. Repomix. https://github.com/yamadashy/repomix, 2024.
  54. Fangyuan Zhang, Lingling Fan, Sen Chen, Miaoying Cai, Sihan Xu, and Lida Zhao. Does the vulnerability threaten our projects? automated vulnerable api detection for third-party libraries. IEEE Transactions on Software Engineering, 2024. Google Scholar
  55. Ying Zhang, Wenjia Song, Zhengjie Ji, Danfeng Yao, and Na Meng. How Can ChatGPT Support Human Security Testers to Help Mitigate Supply Chain Attacks? . IEEE Transactions on Software Engineering, 52(02):509 - 526, 2025. URL: https://doi.org/10.1109/TSE.2025.3632765.
  56. Ying Zhang, Ya Xiao, Md Mahir Asef Kabir, Danfeng (Daphne) Yao, and Na Meng. Example-based vulnerability detection and repair in java code. In Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension, ICPC '22, pages 190-201, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3524610.3527895.
  57. Xin Zhou, Ting Zhang, and David Lo. Large language model for vulnerability detection: Emerging results and future directions. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, pages 47-51, 2024. URL: https://doi.org/10.1145/3639476.3639762.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact