Document Open Access Logo

WebGlitch: A Randomised Testing Tool for the WebGPU API (Experience Paper)

Authors Matthew K. L. Wong , Alastair F. Donaldson

PDF HTML 5

Files

Thumbnail PDF
PDF
LIPIcs.ECOOP.2025.39.pdf
  • Filesize: 1.13 MB
  • 26 pages
HTML5 Logo HTML (experimental)
LIPIcs.ECOOP.2025.39.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
  • Software and its engineering → Object oriented languages
Keywords
  • Fuzzing
  • WebGPU
  • WGSL
  • API
  • shaders

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

We report on our experience designing a new technique and tool for fuzzing implementations of WebGPU, a W3C standard JavaScript API for in-browser GPU computing. We also report on our experience using our WebGlitch tool to test industrial-strength implementations of WebGPU, leading to the discovery of numerous bugs. WebGPU enables programmatic access to a device’s graphics processing unit (GPU) for in-browser GPU computing, and is being implemented by Google, Mozilla and Apple for inclusion in all of the major web browsers. Guaranteeing the security and reliability of WebGPU is crucial to avoid wide-reaching browser security vulnerabilities and to facilitate portability by ensuring uniform behaviour across different platforms. To that end - inspired by randomised compiler testing techniques - our approach to fuzzing creates random, valid-by-construction programs by continuously selecting a WebGPU API function, then recursively generating all requirements necessary for that API call to be valid based on careful modelling of the API specification. This is implemented as a new open source tool, WebGlitch, which we designed in consultation with engineers at Google who work on the Chrome WebGPU implementation. WebGlitch identifies bugs through sanitiser-boosted crash oracles, differential testing, and by identifying cases where valid-by-construction API calls lead to runtime errors. We present an evaluation showing that WebGlitch can find bugs missed by an existing WebGPU fuzzer, wg-fuzz, and across the broader WebGPU ecosystem: to date, WebGlitch has found 24 previously-unknown bugs (15 fixed so far in response to our reports). Among these, 17 bugs affected WebGPU implementations from Google, Mozilla, and the Deno project. WebGlitch found an additional 4 bugs in the shader compilers used by the graphics APIs that WebGPU interfaces with. The remaining 3 bugs affect the widely-used JavaScript runtimes Node.js and Deno. Fuzzing with WebGlitch also led us to identify an ambiguity in the specification of the WebGPU shading language, for which we proposed an amendment that was accepted by W3C and which has been adopted in the latest version of the specification. Analysing the line coverage of a WebGPU implementation by WebGlitch-generated programs revealed that WebGlitch covers code missed by wg-fuzz and the official conformance test suite. Our hope is that this report on the design of WebGlitch and its deployment in practice will be useful for practitioners and researchers interested in using API fuzzing to improve the reliability of industrial codebases.

Cite As Get BibTex

Matthew K. L. Wong and Alastair F. Donaldson. WebGlitch: A Randomised Testing Tool for the WebGPU API (Experience Paper). In 39th European Conference on Object-Oriented Programming (ECOOP 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 333, pp. 39:1-39:26, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.ECOOP.2025.39

Author Details

Matthew K. L. Wong
  • Department of Computing, Imperial College London, UK
Alastair F. Donaldson
  • Department of Computing, Imperial College London, UK

Funding

This work was supported by EPSRC grant EP/R006865/1 and gift funding from Google.

Acknowledgements

We are grateful to Corentin Wallez, Alan Baker and David Neto at Google for their input during the design and deployment of WebGlitch, and to Daniel Simols for assistance in using the wg-fuzz tool.

Supplementary Materials

References

  1. Anon. Automated tests fail on macOS 10.13, 2024. Accessed 23rd April 2025. URL: https://github.com/gfx-rs/wgpu/issues/4632.
  2. Apple. Metal, 2024. Accessed 23rd April 2025. URL: https://developer.apple.com/documentation/metal.
  3. Apple. Metal shading language specification, 2024. Accessed 23rd April 2025. URL: https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf.
  4. Abhishek Arya and Oliver Chang. ClusterFuzz: Fuzzing at Google scale. In Black Hat Europe 2019, 2019. Accessed 23rd April 2025. URL: https://i.blackhat.com/eu-19/Wednesday/eu-19-Arya-ClusterFuzz-Fuzzing-At-Google-Scale.pdf.
  5. Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. RESTler: stateful REST API fuzzing. In Joanne M. Atlee, Tevfik Bultan, and Jon Whittle, editors, Proceedings of the 41st International Conference on Software Engineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pages 748-758. IEEE / ACM, 2019. URL: https://doi.org/10.1109/ICSE.2019.00083.
  6. Lukas Bernhard, Nico Schiller, Moritz Schloegel, Nils Bars, and Thorsten Holz. DarthShader: Fuzzing WebGPU shader translators & compilers. In Bo Luo, Xiaojing Liao, Jun Xu, Engin Kirda, and David Lie, editors, Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS 2024, Salt Lake City, UT, USA, October 14-18, 2024, pages 690-704. ACM, 2024. URL: https://doi.org/10.1145/3658644.3690209.
  7. T.Y. Chen, S.C. Cheung, and S.M. Yiu. Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology, 1998. Google Scholar
  8. Jon Davis. Release notes for Safari Technology Preview 185, 2024. Accessed 23rd April 2025. URL: https://www.webkit.org/blog/14885/release-notes-for-safari-technology-preview-185/.
  9. Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In René Just and Gordon Fraser, editors, Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, Seattle, WA, USA, July 17-21, 2023, pages 423-435. ACM, 2023. URL: https://doi.org/10.1145/3597926.3598067.
  10. Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pages 70:1-70:13. ACM, 2024. URL: https://doi.org/10.1145/3597503.3623343.
  11. Deno. Deno, 2024. Accessed 23rd April 2025. URL: https://github.com/denoland/deno.
  12. Zhen Yu Ding and Claire Le Goues. An empirical study of OSS-fuzz bugs. In 18th IEEE/ACM International Conference on Mining Software Repositories, MSR 2021, Madrid, Spain, May 17-19, 2021, pages 131-142. IEEE, 2021. URL: https://doi.org/10.1109/MSR52588.2021.00026.
  13. Alastair F. Donaldson, Ben Clayton, Ryan Harrison, Hasan Mohsin, David Neto, Vasyl Teliman, and Hana Watson. Industrial deployment of compiler fuzzing techniques for two GPU shading languages. In IEEE Conference on Software Testing, Verification and Validation, ICST 2023, Dublin, Ireland, April 16-20, 2023, pages 374-385. IEEE, 2023. URL: https://doi.org/10.1109/ICST57152.2023.00042.
  14. Alastair F. Donaldson, Hugues Evrard, Andrei Lascu, and Paul Thomson. Automated testing of graphics shader compilers. PACMPL, 1(OOPSLA):93:1-93:29, 2017. URL: https://doi.org/10.1145/3133917.
  15. Alastair F. Donaldson, Hugues Evrard, and Paul Thomson. Putting randomized compiler testing into production (experience report). In Robert Hirschfeld and Tobias Pape, editors, 34th European Conference on Object-Oriented Programming, ECOOP 2020, November 15-17, 2020, Berlin, Germany (Virtual Conference), volume 166 of LIPIcs, pages 22:1-22:29. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2020. URL: https://doi.org/10.4230/LIPICS.ECOOP.2020.22.
  16. Alastair F. Donaldson, Dilan Sheth, Jean-Baptiste Tristan, and Alex Usher. Randomised testing of the compiler for a verification-aware programming language. In IEEE Conference on Software Testing, Verification and Validation, ICST 2024, Toronto, ON, Canada, May 27-31, 2024, pages 407-418. IEEE, 2024. URL: https://doi.org/10.1109/ICST60714.2024.00044.
  17. Alastair F. Donaldson, Paul Thomson, Vasyl Teliman, Stefano Milizia, André Perez Maselco, and Antoni Karpinski. Test-case reduction and deduplication almost for free with transformation-based compiler testing. In Stephen N. Freund and Eran Yahav, editors, PLDI '21: 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, Virtual Event, Canada, June 20-25, 20211, pages 1017-1032. ACM, 2021. URL: https://doi.org/10.1145/3453483.3454092.
  18. Benjamin Fyre. Test failures on Dx12/AMD, 2024. Accessed 23rd April 2025. URL: https://github.com/gfx-rs/wgpu/issues/6830.
  19. Google. Dawn, 2024. Accessed 23rd April 2025. URL: https://github.com/google/dawn.
  20. Google. Sanitizers, 2024. Accessed 23rd April 2025. URL: https://github.com/google/sanitizers.
  21. Harrison Green and Thanassis Avgerinos. Graphfuzz: Library API fuzzing with lifetime-aware dataflow graphs. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pages 1070-1081. ACM, 2022. URL: https://doi.org/10.1145/3510003.3510228.
  22. Kyriakos K. Ispoglou, Daniel Austin, Vishwath Mohan, and Mathias Payer. FuzzGen: Automatic fuzzer generation. In Srdjan Capkun and Franziska Roesner, editors, 29th USENIX Security Symposium, USENIX Security 2020, August 12-14, 2020, pages 2271-2287. USENIX Association, 2020. Accessed 23rd April 2025. URL: https://www.usenix.org/system/files/sec20-ispoglou.pdf.
  23. Jianfeng Jiang, Hui Xu, and Yangfan Zhou. RULF: Rust library fuzzing via API dependency graph traversal. In 36th IEEE/ACM International Conference on Automated Software Engineering, ASE 2021, Melbourne, Australia, November 15-19, 2021, pages 581-592. IEEE, 2021. URL: https://doi.org/10.1109/ASE51524.2021.9678813.
  24. Khronos Group. The OpenGL shading language, version 4.60.8, 2023. Accessed 23rd April 2025. URL: https://registry.khronos.org/OpenGL/specs/gl/GLSLangSpec.4.60.pdf.
  25. Khronos Group. Khronos Vulkan Registry, 2024. Accessed 23rd April 2025. URL: https://registry.khronos.org/vulkan/.
  26. Khronos Group. SPIR-V specification, 2024. Accessed 23rd April 2025. URL: https://registry.khronos.org/SPIR-V/specs/unified1/SPIRV.html.
  27. Khronos Group. WebGL specification, 2024. Accessed 23rd April 2025. URL: https://registry.khronos.org/webgl/specs/latest/1.0/.
  28. Bastien Lecoeur, Hasan Mohsin, and Alastair F. Donaldson. Program reconditioning: Avoiding undefined behaviour when finding and reducing compiler bugs. Proc. ACM Program. Lang., 7(PLDI):1801-1825, 2023. URL: https://doi.org/10.1145/3591294.
  29. LLVM Compiler Infrastructure. libFuzzer – a library for coverage-guided fuzz testing, 2024. Accessed 23rd April 2025. URL: http://llvm.org/docs/LibFuzzer.html.
  30. LLVM Project. llvm-cov - emit coverage information, 2024. Accessed 23rd April 2025. URL: https://llvm.org/docs/CommandGuide/llvm-cov.html.
  31. Chenyang Lyu, Jiacheng Xu, Shouling Ji, Xuhong Zhang, Qinying Wang, Binbin Zhao, Gaoning Pan, Wei Cao, Peng Cheng, and Raheem Beyah. MINER: A hybrid data-driven approach for REST API fuzzing. In Joseph A. Calandrino and Carmela Troncoso, editors, 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023, pages 4517-4534. USENIX Association, 2023. Accessed: 23rd April 2025. URL: https://www.usenix.org/system/files/usenixsecurity23-lyu.pdf.
  32. William M. McKeeman. Differential testing for software. Digit. Tech. J., 10(1):100-107, 1998. Accessed: 23rd April 2025. URL: http://www.dtjcd.vmsresource.org.uk/pdfs/dtj_v10-01_1998.pdf.
  33. Mesa 3D project. LLVMpipe, 2024. Accessed 23rd April 2025. URL: https://docs.mesa3d.org/drivers/llvmpipe.html.
  34. Microsoft. Effect-Compiler Tool, 2020. Accessed 23rd April 2025. URL: https://learn.microsoft.com/en-us/windows/win32/direct3dtools/fxc.
  35. Microsoft. DirectX Shader Compiler, 2024. Accessed 23rd April 2025. URL: https://github.com/microsoft/DirectXShaderCompiler.
  36. Microsoft Corporation. GPU-based validation and the Direct3D 12 debug layer, 2021. Accessed 23rd April 2025. URL: https://learn.microsoft.com/en-us/windows/win32/direct3d12/using-d3d12-debug-layer-gpu-based-validation.
  37. Microsoft Corporation. DirectX graphics and gaming, 2022. Accessed 23rd April 2025. URL: https://learn.microsoft.com/en-us/windows/win32/directx.
  38. Microsoft Corporation. HLSL specifications, 2024. Accessed 23rd April 2025. URL: https://github.com/microsoft/hlsl-specs.
  39. David Neto and Alan Baker. Personal correspondence, July 2024. Google Scholar
  40. NIST National Vulnerability Database. CVE-2016-2824, 2018. Accessed 23rd April 2025. URL: https://nvd.nist.gov/vuln/detail/CVE-2016-2824.
  41. NIST National Vulnerability Database. CVE-2012-3968, 2020. Accessed 23rd April 2025. URL: https://nvd.nist.gov/vuln/detail/CVE-2012-3968.
  42. NIST National Vulnerability Database. CVE-2017-5112, 2023. Accessed 23rd April 2025. URL: https://nvd.nist.gov/vuln/detail/CVE-2017-5112.
  43. Carlos Pacheco, Shuvendu K. Lahiri, Michael D. Ernst, and Thomas Ball. Feedback-directed random test generation. In 29th International Conference on Software Engineering (ICSE 2007), Minneapolis, MN, USA, May 20-26, 2007, pages 75-84. IEEE Computer Society, 2007. URL: https://doi.org/10.1109/ICSE.2007.37.
  44. Hui Peng, Zhihao Yao, Ardalan Amiri Sani, Dave Tian, and Mathias Payer. GLeeFuzz: Fuzzing WebGL through error message guided mutation. In Joseph A. Calandrino and Carmela Troncoso, editors, 32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023, pages 1883-1899. USENIX Association, 2023. Accessed 23rd April 2025. URL: https://www.usenix.org/system/files/usenixsecurity23-peng.pdf.
  45. Rust Graphics Mages. wgpu, 2024. Accessed 23rd April 2025. URL: https://github.com/gfx-rs/wgpu?tab=readme-ov-file.
  46. Hynek Schlawack. Surprising consequences of macOS’s environment variable sanitization, 2023. Accessed 23rd April 2025. URL: https://hynek.me/articles/macos-dyld-env/.
  47. Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. AddressSanitizer: A fast address sanity checker. In Gernot Heiser and Wilson C. Hsieh, editors, Proceedings of the 2012 USENIX Annual Technical Conference, USENIX ATC 2012, Boston, MA, USA, June 13-15, 2012, pages 309-318. USENIX Association, 2012. Accessed 23rd April 2025. URL: https://www.usenix.org/system/files/conference/atc12/atc12-final39.pdf.
  48. Daniel Simols. wg-fuzz, 2024. Accessed 23rd April 2025. URL: https://github.com/wg-fuzz/wg-fuzz.
  49. Brendon Tiszka, Chris Bookholt, and Matthew Riley. WebGPU technical report. Technical report, Google, 2023. Accessed 23rd April 2025. URL: https://chromium.googlesource.com/chromium/src/+/main/docs/security/research/graphics/webgpu_technical_report.md?pli=1#.
  50. W3C group for GPU web standards. Conformance Test Suite, 2024. Accessed 23rd April 2025. URL: https://github.com/gpuweb/cts.
  51. W3C group for GPU web standards. Implementation status, 2024. Accessed 23rd April 2025. URL: https://github.com/gpuweb/gpuweb/wiki/Implementation-Status.
  52. Matthew Wong. Add extra example for operations involving abstract numerics, 2024. Accessed 23rd April 2025. URL: https://github.com/gpuweb/gpuweb/pull/4795.
  53. Matthew Wong. WebGlitch, 2024. Accessed 23rd April 2025. URL: https://github.com/matthew-wong1/WebGlitch.
  54. Matthew Wong. WebGPU adapter.info call fails after device creation, 2024. Accessed 23rd April 2025. URL: https://github.com/denoland/deno/issues/25874.
  55. Matthew Wong. naga change i32 arithmetic operations to use wrapping instead of checked, 2025. Accessed 23rd April 2025. URL: https://github.com/gfx-rs/wgpu/pull/6835.
  56. Matthew Wong. WebGlitch artifact, 2025. Accessed 25th April 2025. URL: https://doi.org/10.5281/zenodo.15281221.
  57. World Wide Web Consortium. WebGPU explainer draft community group report, 2024. Accessed 23rd April 2025. URL: https://gpuweb.github.io/gpuweb/explainer/.
  58. World Wide Web Consortium. WebGPU W3C working draft, 2024. Accessed 23rd April 2025. URL: https://www.w3.org/TR/webgpu/.
  59. World Wide Web Consortium. WebGPU shading language W3C working draft, 4 September 2024, 2024. Accessed 23rd April 2025. URL: https://www.w3.org/TR/2024/WD-WGSL-20240904/.
  60. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. Finding and understanding bugs in C compilers. In Mary W. Hall and David A. Padua, editors, Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2011, San Jose, CA, USA, June 4-8, 2011, pages 283-294. ACM, 2011. URL: https://doi.org/10.1145/1993498.1993532.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact