Document Open Access Logo

FDRRestTest: Feedback-Driven Logical Testing for REST APIs

Authors Yutong Huo , Dongcheng Li

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.20.pdf
  • Filesize: 0.93 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
Keywords
  • REST API testing
  • black-box testing
  • logical defects
  • feedback-driven testing
  • oracle-based validation

Metrics

Abstract

Background. REST APIs have become the primary interaction interface for modern software systems, making automated black-box API testing increasingly important. However, existing testing techniques still face three major limitations: over-reliance on dependencies extracted from static specifications, failure to fully utilise failed requests as feedback, and an overemphasis on crash-oriented verification, resulting in insufficient exploration of non-crash-prone logical defects.

Aims. We design, implement and evaluate a feedback-driven black-box testing framework that turns runtime feedback from a passive outcome into an active control signal, maximising logical-defect yield within practical budgets.

Method. We propose FDRRestTest, which integrates OpenAPI-based static semantic bootstrapping, runtime dependency evolution, a 7-category failure diagnosis and repair loop, a multi-dimensional logical oracle engine and a budget-aware utility scheduler in one closed loop. Under a fixed 600-second time budget, we evaluated FDRRestTest against five state-of-the-art baselines (RESTler, EvoMaster, Morest, ARAT-RL, AutoRestTest) on 12 real-world REST services along five protocols: fixed-time cross-tool comparison, fixed-request cross-tool replication, ablation across three dependency profiles, single-host budget sensitivity, and per-service comparison; significance via two-sided paired Wilcoxon signed-rank tests with Holm-Bonferroni correction, magnitude via standardised paired effect sizes.

Results. FDRRestTest improves fault-revealing efficiency (FRE) by up to 6.3% over the strongest baseline, AutoRestTest (0.9878 vs. 0.9294; an upper estimate, as AutoRestTest’s FRE is a post-hoc lower bound), while issuing 6.3-11.8× fewer requests than AutoRestTest across the 12 services (8.3× on Features Service). The +12.2% logical-yield advantage holds on every one of the 12 services (+7.0%-+16.5%, geometric mean +12.4%), and the FRE gain is consistent across all 12 in per-service means and statistically significant (p_{adj} < .005, d_z = 3.73 on logical bug yield, LBY). An ablation across six subjects spanning three dependency profiles confirms no single component accounts for the advantage, the oracle contributing most - removing it drops FRE below the strongest baseline on all six - while a Pareto-scheduling variant is 4.4% below the weighted utility on the most state-rich host. A manual confirmation study (N = 120, stratified over the 236 deduplicated findings of all six tools, 30 per oracle category) confirms 96/120 (80%) as true defects (Cohen’s κ = 0.78), supporting the oracle-finding interpretation of the LBY gain.

Conclusions. Black-box REST API testing heavily benefits from a tightly coupled closed feedback loop. Within this loop, runtime dependency evolution, failure repair, logical-defect-aware validation, and budget-aware scheduling are each necessary; quantifying their interaction would require multi-component removals, which we leave to future work.

Cite As Get BibTex

Yutong Huo and Dongcheng Li. FDRRestTest: Feedback-Driven Logical Testing for REST APIs. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 20:1-20:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.20

Author Details

Yutong Huo
  • China University of Geosciences, Wuhan, China
Dongcheng Li
  • Cal Poly Humboldt, Arcata, CA, USA

Supplementary Materials

References

  1. J. C. Alonso, A. Martin-Lopez, S. Segura, J. Garcia, and A. Ruiz-Cortes. ARTE: Automated generation of realistic test inputs for web APIs. IEEE Transactions on Software Engineering, 49(1):348-363, 2023. URL: https://doi.org/10.1109/TSE.2022.3150618.
  2. Andrea Arcuri. RESTful API automated test case generation with EvoMaster. ACM Transactions on Software Engineering and Methodology, 28(1):3:1-3:37, 2019. URL: https://doi.org/10.1145/3293455.
  3. Andrea Arcuri and Lionel Briand. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability, 24(3):219-250, 2014. URL: https://doi.org/10.1002/stvr.1486.
  4. Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. RESTler: Stateful REST API fuzzing. In Proceedings of the 41st International Conference on Software Engineering, pages 748-758, 2019. URL: https://doi.org/10.1109/ICSE.2019.00083.
  5. Norman Cliff. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin, 114(3):494-509, 1993. URL: https://doi.org/10.1037/0033-2909.114.3.494.
  6. Davide Corradini, Zeno Montolli, Michele Pasqua, and Mariano Ceccato. DeepREST: Automated test case generation for REST APIs exploiting deep reinforcement learning. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 1383-1394, 2024. URL: https://doi.org/10.1145/3691620.3695511.
  7. Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. Automated black-box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability, 32(5):e1808, 2022. URL: https://doi.org/10.1002/stvr.1808.
  8. Alix Decrop, Gilles Perrouin, Mike Papadakis, Xavier Devroey, and Pierre-Yves Schobbens. You can REST now: Automated specification inference and black-box testing of RESTful APIs with large language models. arXiv preprint arXiv:2402.05102, 2024. URL: https://doi.org/10.48550/arXiv.2402.05102.
  9. Sida Deng, Rubing Huang, Man Zhang, Chenhui Cui, Dave Towey, and Rongcun Wang. LRASGen: LLM-based RESTful API specification generation. arXiv preprint arXiv:2504.16833, 2025. Manuscript submitted to ACM. URL: https://doi.org/10.48550/arXiv.2504.16833.
  10. Roy Thomas Fielding. Architectural Styles and the Design of Network-based Software Architectures. Ph.D. dissertation, University of California, Irvine, 2000. URL: https://roy.gbiv.com/pubs/dissertation/fielding_dissertation.pdf.
  11. Amid Golmohammadi, Man Zhang, and Andrea Arcuri. Testing RESTful APIs: A survey. ACM Transactions on Software Engineering and Methodology, 33(1):27:1-27:41, 2023. URL: https://doi.org/10.1145/3617175.
  12. Zac Hatfield-Dodds and Dmitry Dygalo. Deriving semantics-aware fuzzers from web API schemas. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pages 345-346, 2022. URL: https://doi.org/10.1145/3510454.3528637.
  13. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979. URL: https://www.jstor.org/stable/4615733.
  14. Minh-Hieu Huynh, Quoc-Tri Le, Tien N. Nguyen, and Vu Nguyen. Using LLM for mining and testing constraints in API testing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 2486-2487, 2024. URL: https://doi.org/10.1145/3691620.3695341.
  15. Stefan Karlsson, Adnan Čaušević, and Daniel Sundmark. QuickREST: Property-based test generation of OpenAPI-described RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification, pages 131-141, 2020. URL: https://doi.org/10.1109/ICST46399.2020.00023.
  16. Myeongsoo Kim, Davide Corradini, Saurabh Sinha, Alessandro Orso, Michele Pasqua, Rachel Tzoref-Brill, and Mariano Ceccato. Enhancing REST API testing with NLP techniques. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1232-1243, 2023. URL: https://doi.org/10.1145/3597926.3598131.
  17. Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. Adaptive REST API testing with reinforcement learning. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, pages 446-458, 2023. URL: https://doi.org/10.1109/ASE56229.2023.00218.
  18. Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. LlamaRestTest: Effective REST API testing with small language models. Proceedings of the ACM on Software Engineering, 2(FSE):FSE022:1-FSE022:24, 2025. URL: https://doi.org/10.1145/3715737.
  19. Myeongsoo Kim, Tyler Stennett, Dhruv Shah, Saurabh Sinha, and Alessandro Orso. Leveraging large language models to improve REST API testing. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, pages 37-41, 2024. URL: https://doi.org/10.1145/3639476.3639769.
  20. Myeongsoo Kim, Tyler Stennett, Saurabh Sinha, and Alessandro Orso. A multi-agent approach for REST API testing with semantic graphs and LLM-driven inputs. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, ICSE '25, pages 1409-1421. IEEE Press, 2025. URL: https://doi.org/10.1109/ICSE55347.2025.00179.
  21. Myeongsoo Kim, Qi Xin, Saurabh Sinha, and Alessandro Orso. Automated test generation for REST APIs: No time to REST yet. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 289-301, 2022. URL: https://doi.org/10.1145/3533767.3534401.
  22. Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. CODAMOSA: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering, pages 919-931, 2023. URL: https://doi.org/10.1109/ICSE48619.2023.00085.
  23. Jiaxian Lin, Tianyu Li, Yang Chen, Guangsheng Wei, Jiadong Lin, Sen Zhang, and Hui Xu. foREST: A tree-based black-box fuzzing approach for RESTful APIs. In 2023 IEEE 34th International Symposium on Software Reliability Engineering, pages 695-705, 2023. URL: https://doi.org/10.1109/ISSRE59848.2023.00023.
  24. Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. Morest: Model-based RESTful API testing with execution feedback. In Proceedings of the 44th International Conference on Software Engineering, pages 1406-1417, 2022. URL: https://doi.org/10.1145/3510003.3510133.
  25. Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. A catalogue of inter-parameter dependencies in RESTful web APIs. In Service-Oriented Computing, pages 399-414, 2019. URL: https://doi.org/10.1007/978-3-030-33702-5_31.
  26. Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. RESTest: Automated black-box testing of RESTful web APIs. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 682-685, 2021. URL: https://doi.org/10.1145/3460319.3469082.
  27. OpenAPI. OpenAPI standard. https://www.openapis.org, 2023.
  28. Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. ASTER: Natural and multi-language unit test generation with LLMs. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP '25, pages 413-424. IEEE Press, 2025. URL: https://doi.org/10.1109/ICSE-SEIP66354.2025.00042.
  29. Cesare Pautasso, Olaf Zimmermann, and Frank Leymann. RESTful web services vs. "big" web services: Making the right architectural decision. In Proceedings of the 17th International Conference on World Wide Web, pages 805-814, 2008. URL: https://doi.org/10.1145/1367497.1367606.
  30. André Pereira, Bruno Lima, and João Pascoal Faria. ApiTestGenie: Automated API test generation through generative AI. arXiv preprint arXiv:2409.03838, 2024. URL: https://doi.org/10.48550/arXiv.2409.03838.
  31. Leonard Richardson, Mike Amundsen, and Sam Ruby. RESTful Web APIs: Services for a Changing World. O'Reilly Media, 2013. Google Scholar
  32. SmartBear Software. Swagger specification. https://swagger.io/specification/v2/, 2023.
  33. Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. RestTestGen: Automated black-box testing of RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification, pages 142-152, 2020. URL: https://doi.org/10.1109/ICST46399.2020.00024.
  34. Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 50(4):911-936, 2024. URL: https://doi.org/10.1109/TSE.2024.3368208.
  35. Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. URL: https://doi.org/10.2307/3001968.
  36. Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. Combinatorial testing of RESTful APIs. In Proceedings of the 44th International Conference on Software Engineering, pages 426-437, 2022. URL: https://doi.org/10.1145/3510003.3510151.
  37. Ke Zhang, Chenxi Zhang, Chong Wang, Chi Zhang, YaChen Wu, Zhenchang Xing, Yang Liu, Qingshan Li, and Xin Peng. LogiAgent: Automated logical testing for REST systems with LLM-based multi-agents. arXiv preprint arXiv:2503.15079, 2025. URL: https://doi.org/10.48550/arXiv.2503.15079.
  38. Man Zhang and Andrea Arcuri. Open problems in fuzzing RESTful APIs: A comparison of tools. ACM Transactions on Software Engineering and Methodology, 32(6):144:1-144:45, 2023. URL: https://doi.org/10.1145/3597205.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact