Search Results

Documents authored by Grundy, John

Document
Emerging Results, Vision & Reflection Track Paper
DOI: 10.4230/LIPIcs.ESEM.2026.72
Ethical Requirements for Healthcare AI Software Systems: Emerging Results from a Multi-Stakeholder Interview Study

Authors: Yutan Huang, Chetan Arora, Tanjila Kanij, Anuradha Madugalla, and John Grundy

Published in: LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)

Abstract
Healthcare AI software systems raise ethical concerns that are often discussed as high-level principles but are difficult to translate into concrete software engineering requirements. This emerging-results paper reports preliminary findings from a multi-stakeholder interview study on how practitioners experience and prioritise ethical concerns in healthcare AI software systems. We conducted 17 semi-structured interviews with healthcare professionals (n=5), healthcare researchers (n=6), and software engineers (n=6). Using thematic synthesis, we identified three AI use contexts and four recurring challenge categories: model reliability and quality, data quality and processing, human oversight and skills gaps, and institutional and regulatory constraints. Privacy and safety emerged as baseline concerns across roles, while bias, accountability, and transparency varied depending on practitioners' responsibilities and use contexts. Based on these preliminary findings, we propose role-specific patterns as lightweight requirements artefacts to translate practitioner priorities into actionable software engineering practices across requirements elicitation, system design, testing, and auditing. We present these patterns as an initial artefact for community feedback and as a first step toward a broader approach to ethical requirements engineering (RE) for healthcare AI software systems. Future work will validate and refine the patterns using a larger, more balanced sample, evaluate their usefulness through practitioner workshops, and develop a reusable catalogue with guidance on traceability, verification, and auditing. Long-term, our goal is to support healthcare AI teams in moving from abstract ethical principles to testable requirements that can be integrated into existing development and governance workflows.
Cite as

Yutan Huang, Chetan Arora, Tanjila Kanij, Anuradha Madugalla, and John Grundy. Ethical Requirements for Healthcare AI Software Systems: Emerging Results from a Multi-Stakeholder Interview Study. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 72:1-72:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026)

Copy BibTex To Clipboard

@InProceedings{huang_et_al:LIPIcs.ESEM.2026.72,
  author =	{Huang, Yutan and Arora, Chetan and Kanij, Tanjila and Madugalla, Anuradha and Grundy, John},
  title =	{{Ethical Requirements for Healthcare AI Software Systems: Emerging Results from a Multi-Stakeholder Interview Study}},
  booktitle =	{20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)},
  pages =	{72:1--72:14},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-450-5},
  ISSN =	{1868-8969},
  year =	{2026},
  volume =	{394},
  editor =	{Feldt, Robert and Paasivaara, Maria and Mendez, Daniel and Wagner, Stefan and Bar\'{o}n, Marvin Mu\~{n}oz},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.72},
  URN =		{urn:nbn:de:0030-drops-280406},
  doi =		{10.4230/LIPIcs.ESEM.2026.72},
  annote =	{Keywords: Healthcare AI software systems, ethical requirements, requirements engineering, empirical software engineering, thematic synthesis, practitioner interviews}
}
Document
Software Engineering in Practice Track Paper
DOI: 10.4230/LIPIcs.ESEM.2026.87
AI Ethics to Requirements Practice: Building and Evaluating the HealthAI Ethics Assistant

Authors: Yutan Huang, Jingfan Chen, Fanyu Wang, Chetan Arora, John Grundy, and Cheng Zhang

Published in: LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)

Abstract
Healthcare AI software teams face growing pressure to translate ethical principles and regulatory obligations into concrete requirements, design decisions, and assurance evidence. Day-to-day requirements engineering (RE) practice still lacks lightweight tool support for connecting abstract frameworks, such as the EU AI Act and the NIST AI Risk Management Framework, to actionable requirements work. This paper builds on a two-phase research project. In the first phase, we conducted an empirical interview study with healthcare AI practitioners spanning clinicians as end users of HealthAI systems and software engineers as developers of HealthAI software systems, which surfaced recurring concerns around transparency, accountability, and the difficulty of mapping regulatory obligations to concrete engineering tasks. Informed by those findings, in the second phase we designed, implemented, and conducted an initial evaluation of the HealthAI Ethics Assistant, an AI-enabled RE tool for healthcare AI software systems. The tool supports practitioners in generating, validating, and comparing ethical requirements through a structured Create-Validate-Compare workflow. It was implemented as a full-stack web application and grounded in a structured knowledge base combining regulatory guidance (EU AI Act, NIST AI RMF) with practitioner concerns surfaced by the interview study. The work was conducted in close collaboration with an R&D engineer at a medical device manufacturer, who contributed to industrial problem framing and participated as the first practitioner evaluator in the case study reported here. Our initial evaluation is exploratory, involving a single case study, and used the Technology Acceptance Model and the System Usability Scale. Traceable regulatory references were rated as the most valuable and trustworthy feature, highlighting the importance of explainability and evidence support in compliance-oriented requirements work. The main adoption challenge was not basic usability, but fitting the tool into existing engineering and regulatory workflows. We derive practice-oriented lessons for designing AI-enabled requirements tools in regulated domains: ground LLM outputs in explicit compliance sources, support multiple practitioner perspectives, make generated requirements reviewable rather than authoritative, and align tool use with existing assurance processes. These early results suggest that AI-enabled assistants can help bridge ethical AI principles and practical requirements engineering when designed as auditable, human-in-the-loop support tools rather than autonomous compliance solutions.
Cite as

Yutan Huang, Jingfan Chen, Fanyu Wang, Chetan Arora, John Grundy, and Cheng Zhang. AI Ethics to Requirements Practice: Building and Evaluating the HealthAI Ethics Assistant. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 87:1-87:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026)

Copy BibTex To Clipboard

@InProceedings{huang_et_al:LIPIcs.ESEM.2026.87,
  author =	{Huang, Yutan and Chen, Jingfan and Wang, Fanyu and Arora, Chetan and Grundy, John and Zhang, Cheng},
  title =	{{AI Ethics to Requirements Practice: Building and Evaluating the HealthAI Ethics Assistant}},
  booktitle =	{20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)},
  pages =	{87:1--87:13},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-450-5},
  ISSN =	{1868-8969},
  year =	{2026},
  volume =	{394},
  editor =	{Feldt, Robert and Paasivaara, Maria and Mendez, Daniel and Wagner, Stefan and Bar\'{o}n, Marvin Mu\~{n}oz},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.87},
  URN =		{urn:nbn:de:0030-drops-280555},
  doi =		{10.4230/LIPIcs.ESEM.2026.87},
  annote =	{Keywords: Requirements Engineering, Software Engineering, AI ethics, Healthcare, Large Language Models, Regulatory Compliance.}
}

Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact