Document Open Access Logo

Unsupervised Multimodal Learning for Fault Diagnosis and Prognosis - Application to Radiotherapy Systems (PhD Panel)

Authors Kélian Poujade , Louise Travé-Massuyès , Jérémy Pirard , Laure Vieillevigne

PDF

File

Thumbnail PDF
PDF
OASIcs.DX.2025.16.pdf
  • Filesize: 9.31 MB
  • 17 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Unsupervised learning
Keywords
  • Artificial Intelligence
  • Diagnosis
  • Prognosis
  • Radiotherapy machines

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Modern complex systems, such as radiotherapy machines, require robust strategies for fault detection, diagnosis, and prognosis to ensure operational continuity and patient safety. While data-driven methods have gained traction, few studies address diagnostic and prognostic tasks using multimodal operational data under unsupervised or semi-supervised learning settings. This gap is particularly critical given the scarcity of labeled failure data in real-world environments. This work aims to design a unified approach for fault detection, diagnosis, and prognosis using multimodal data in the absence of complete labeling. To this end, autoencoders (AEs) are employed due to their suitability for unsupervised and self-supervised learning, flexibility in handling heterogeneous data, and ability to construct latent representations optimized for various downstream tasks. A specific implementation based on a Long Short-Term Memory β-Variational Autoencoder (LSTM-β-VAE) was developed to detect anomalies in machine logs. This framework is applied to TomoTherapy® systems - a highly complex and under-explored use case within the radiotherapy domain. Initial results demonstrate strong anomaly detection performance on both a public benchmark dataset (HDFS) and a proprietary dataset derived from real-world TomoTherapy® machine faults. Beyond methodology, the paper includes a concise literature review of multimodal learning and data-driven diagnosis and prognosis with a focus on AEs. Based on this review, key research directions are identified for the continuation of the thesis, especially the integration of explainable AI as a means to enhance diagnosis capabilities in the absence of labeled faults.

Cite As Get BibTex

Kélian Poujade, Louise Travé-Massuyès, Jérémy Pirard, and Laure Vieillevigne. Unsupervised Multimodal Learning for Fault Diagnosis and Prognosis - Application to Radiotherapy Systems (PhD Panel). In 36th International Conference on Principles of Diagnosis and Resilient Systems (DX 2025). Open Access Series in Informatics (OASIcs), Volume 136, pp. 16:1-16:17, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.DX.2025.16

Author Details

Kélian Poujade
  • Université de Toulouse, Oncopole Claudius Regaud, Institut Universitaire du Cancer de Toulouse (IUCT), France
  • Université de Toulouse, CNRS, INSERM, Centre de Recherches en Cancérologie de Toulouse (CRCT), France
Louise Travé-Massuyès
  • Laboratoire d'Analyse et d'Architecture des Systèmes (LAAS-CNRS), Université de Toulouse, CNRS, France
Jérémy Pirard
  • Airbus, Toulouse, France
Laure Vieillevigne
  • Université de Toulouse, Oncopole Claudius Regaud, Institut Universitaire du Cancer de Toulouse (IUCT), France
  • Université de Toulouse, CNRS, INSERM, Centre de Recherches en Cancérologie de Toulouse (CRCT), France

Funding

This research was supported by Accuray Inc as part of the doctoral funding. It has also benefited from the AI Interdisciplinary Institute ANITI funded by the France 2030 program under the Grant agreement n°ANR-23-IACL-0002.

Acknowledgements

This work is the result of a collaborative tripartite agreement involving IUCT-Oncopole, Airbus, and Accuray Inc.

References

  1. Charles M Able, Alan H Baydush, Callistus Nguyen, Jacob Gersh, Alois Ndlovu, Igor Rebo, Jeremy Booth, Mario Perez, Benjamin Sintay, and Michael T Munley. A model for preemptive maintenance of medical linear accelerators—predictive maintenance. Radiation Oncology, 11:1-9, 2016. Google Scholar
  2. Christina Elizabeth Agnew, Sergio Esteve, Glenn Whitten, and William Little. Reducing treatment machine downtime with a preventative mlc maintenance procedure. Physica Medica, 85:1-7, 2021. Google Scholar
  3. Diana Laura Aguilar, Miguel Angel Medina-Pérez, Octavio Loyola-Gonzalez, Kim-Kwang Raymond Choo, and Edoardo Bucheli-Susarrey. Towards an interpretable autoencoder: A decision-tree-based autoencoder and its application in anomaly detection. IEEE transactions on dependable and secure computing, 20(2):1048-1059, 2022. URL: https://doi.org/10.1109/TDSC.2022.3148331.
  4. Nima Amini and Qinqin Zhu. Fault detection and diagnosis with a novel source-aware autoencoder and deep residual neural network. Neurocomputing, 488:618-633, 2022. URL: https://doi.org/10.1016/J.NEUCOM.2021.11.067.
  5. Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021. URL: https://arxiv.org/abs/2107.07511.
  6. Liat Antwarg, Ronnie Mindlin Miller, Bracha Shapira, and Lior Rokach. Explaining anomalies detected by autoencoders using shapley additive explanations. Expert systems with applications, 186:115736, 2021. URL: https://doi.org/10.1016/J.ESWA.2021.115736.
  7. Pietro Baroni, Gianfranco Lamperti, Paolo Pogliano, and Marina Zanella. Diagnosis of large active systems. Artificial Intelligence, 110(1):135-183, 1999. URL: https://doi.org/10.1016/S0004-3702(99)00019-3.
  8. Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000. URL: https://doi.org/10.1145/342009.335388.
  9. Davide Cacciarelli and Murat Kulahci. A novel fault detection and diagnosis approach based on orthogonal autoencoders. Computers & Chemical Engineering, 163:107853, 2022. URL: https://doi.org/10.1016/J.COMPCHEMENG.2022.107853.
  10. Sarath Chandar, Mitesh M Khapra, Hugo Larochelle, and Balaraman Ravindran. Correlational neural networks. Neural computation, 28(2):257-285, 2016. URL: https://doi.org/10.1162/NECO_A_00801.
  11. Zhelun Chen, Zheng O’Neill, Jin Wen, Ojas Pradhan, Tao Yang, Xing Lu, Guanjing Lin, Shohei Miyata, Seungjae Lee, Chou Shen, et al. A review of data-driven fault detection and diagnostics for building hvac systems. Applied Energy, 339:121030, 2023. Google Scholar
  12. Stéphan Clémençon and Jérémie Jakubowicz. Scoring anomalies: a m-estimation formulation. In Artificial Intelligence and Statistics, pages 659-667. PMLR, 2013. URL: http://proceedings.mlr.press/v31/clemencon13a.html.
  13. Johan De Kleer and Brian C Williams. Diagnosing multiple faults. Artificial intelligence, 32(1):97-130, 1987. URL: https://doi.org/10.1016/0004-3702(87)90063-4.
  14. Ingeborg de Pater and Mihaela Mitici. Developing health indicators and rul prognostics for systems with few failure instances and varying operating conditions using a lstm autoencoder. Engineering Applications of Artificial Intelligence, 117:105582, 2023. URL: https://doi.org/10.1016/J.ENGAPPAI.2022.105582.
  15. Francesco Di Clemente, Matteo Scialpi, and Michał Bejger. Explainable autoencoder for neutron star dense matter parameter estimation. Machine Learning: Science and Technology, 2025. Google Scholar
  16. Tai Dou, Benjamin Clasie, Nicolas Depauw, Tim Shen, Robert Brett, Hsiao-Ming Lu, Jacob B Flanz, and Kyung-Wook Jee. A deep lstm autoencoder-based framework for predictive maintenance of a proton radiotherapy delivery system. Artificial Intelligence in Medicine, 132:102387, 2022. URL: https://doi.org/10.1016/J.ARTMED.2022.102387.
  17. Wenjun Feng, Xin Wang, Donglin Cao, and Dazhen Lin. An autoencoder-based self-supervised learning for multimodal sentiment analysis. Information Sciences, 675:120682, 2024. URL: https://doi.org/10.1016/J.INS.2024.120682.
  18. José A González Ferreira, Javier Jaén Olasolo, Ignacio Azinovic, and Branislav Jeremic. Effect of radiotherapy delay in overall treatment time on local control and survival in head and neck cancer: review of the literature. Reports of Practical Oncology and Radiotherapy, 20(5):328-339, 2015. Google Scholar
  19. Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205.14204, 2022. URL: https://doi.org/10.48550/arXiv.2205.14204.
  20. Nicolas Goix. How to evaluate the quality of unsupervised anomaly detection algorithms? arXiv preprint arXiv:1607.01152, 2016. URL: https://arxiv.org/abs/1607.01152.
  21. Anselm Haselhoff, Kevin Trelenberg, Fabian Küppers, and Jonas Schneider. The gaussian discriminant variational autoencoder (gdvae): A self-explainable model with counterfactual explanations. In European Conference on Computer Vision, pages 305-322. Springer, 2024. URL: https://doi.org/10.1007/978-3-031-73668-1_18.
  22. Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R Lyu. Drain: An online log parsing approach with fixed depth tree. In 2017 IEEE international conference on web services (ICWS), pages 33-40. IEEE, 2017. URL: https://doi.org/10.1109/ICWS.2017.13.
  23. Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2017. Google Scholar
  24. Tung Kieu, Bin Yang, Chenjuan Guo, Christian S Jensen, Yan Zhao, Feiteng Huang, and Kai Zheng. Robust and explainable autoencoders for unsupervised time series outlier detection. In 2022 IEEE 38th International conference on data engineering (ICDE), pages 3038-3050. IEEE, 2022. URL: https://doi.org/10.1109/ICDE53745.2022.00273.
  25. Håvard Kvamme and Ørnulf Borgan. Continuous and discrete-time survival prediction with neural networks. Lifetime data analysis, 27(4):710-736, 2021. Google Scholar
  26. Qing Li, Yang Yu, Pathum Kossinna, Theodore Lun, Wenyuan Liao, and Qingrun Zhang. Xa4c: explainable representation learning via autoencoders revealing critical genes. PLOS Computational Biology, 19(10):e1011476, 2023. URL: https://doi.org/10.1371/JOURNAL.PCBI.1011476.
  27. Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. Isolation forest. In 2008 eighth ieee international conference on data mining, pages 413-422. IEEE, 2008. URL: https://doi.org/10.1109/ICDM.2008.17.
  28. Han Liu, Jianzhong Zhou, Yang Zheng, Wei Jiang, and Yuncheng Zhang. Fault diagnosis of rolling bearings with recurrent neural network-based autoencoders. ISA transactions, 77:167-178, 2018. Google Scholar
  29. Lang Liu, Ying Zheng, and Shaojun Liang. Variable-wise stacked temporal autoencoder for intelligent fault diagnosis of industrial systems. IEEE Transactions on Industrial Informatics, 2024. Google Scholar
  30. Wenqian Liu, Runze Li, Meng Zheng, Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard J Radke, and Octavia Camps. Towards visually explaining variational autoencoders. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8642-8651, 2020. Google Scholar
  31. Zhao-Hua Liu, Xu-Dong Meng, Hua-Liang Wei, Liang Chen, Bi-Liang Lu, Zhen-Heng Wang, and Lei Chen. A regularized lstm method for predicting remaining useful life of rolling bearings. International Journal of Automation and Computing, 18:581-593, 2021. URL: https://doi.org/10.1007/S11633-020-1276-6.
  32. Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017. Google Scholar
  33. Meng Ma, Chuang Sun, and Xuefeng Chen. Deep coupling autoencoder for fault diagnosis with multimodal sensory data. IEEE Transactions on Industrial Informatics, 14(3):1137-1145, 2018. URL: https://doi.org/10.1109/TII.2018.2793246.
  34. Min Ma, Chenbin Liu, Ran Wei, Bin Liang, and Jianrong Dai. Predicting machine’s performance record using the stacked long short-term memory (lstm) neural networks. Journal of Applied Clinical Medical Physics, 23(3):e13558, 2022. Google Scholar
  35. T Rockwell Mackie, John Balog, Ken Ruchala, Dave Shepard, Stacy Aldridge, Ed Fitchard, Paul Reckwerdt, Gustavo Olivera, Todd McNutt, and Minesh Mehta. Tomotherapy. In Seminars in Radiation Oncology, volume 9, pages 108-117. Elsevier, 1999. Google Scholar
  36. Miguel Martinez-Garcia, Yu Zhang, Jiafu Wan, and Jason Mcginty. Visually interpretable profile extraction with an autoencoder for health monitoring of industrial systems. In 2019 IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM), pages 649-654. IEEE, 2019. Google Scholar
  37. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. Google Scholar
  38. Quoc Phong Nguyen, Kar Wai Lim, Dinil Mon Divakaran, Kian Hsiang Low, and Mun Choon Chan. Gee: A gradient-based explainable variational autoencoder for network anomaly detection. In 2019 IEEE Conference on Communications and Network Security (CNS), pages 91-99. IEEE, 2019. URL: https://doi.org/10.1109/CNS.2019.8802833.
  39. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. Google Scholar
  40. Bernhard Peischl and Franz Wotawa. Model-based diagnosis or reasoning from first principles. IEEE intelligent systems, 18(3):32-37, 2005. Google Scholar
  41. Ingo Pill and Johan De Kleer. Challenges for model-based diagnosis. In 35th International Conference on Principles of Diagnosis and Resilient Systems (DX 2024), pages 6-1. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2024. URL: https://doi.org/10.4230/OASIcs.DX.2024.6.
  42. Kélian Poujade, Louise Travé-Massuyès, Jérémy Pirard, and Laure Vieillevigne. B-variational autoencoder based anomaly detection in log data - application to radiotherapy systems. Preprint, HAL archive, 2025. URL: https://hal.science/hal-05209127.
  43. Zhi Qiu, Shanfei Fan, Haibo Liang, and Jincai Liu. Multimodal fusion fault diagnosis method under noise interference. Applied Acoustics, 228:110301, 2025. Google Scholar
  44. Raymond Reiter. A theory of diagnosis from first principles. Artificial intelligence, 32(1):57-95, 1987. URL: https://doi.org/10.1016/0004-3702(87)90062-2.
  45. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016. Google Scholar
  46. Lucas Robinet, Ahmad Berjaoui, Ziad Kheil, and Elizabeth Cohen-Jonathan Moyal. Drim: Learning disentangled representations from incomplete multimodal healthcare data. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 163-173. Springer, 2024. URL: https://doi.org/10.1007/978-3-031-72384-1_16.
  47. Atma Ram Sahu, Sanjay Kumar Palei, and Aishwarya Mishra. Data-driven fault diagnosis approaches for industrial equipment: A review. Expert Systems, 41(2):e13360, 2024. URL: https://doi.org/10.1111/EXSY.13360.
  48. Bernhard Schölkopf, Robert C Williamson, Alex Smola, John Shawe-Taylor, and John Platt. Support vector method for novelty detection. Advances in neural information processing systems, 12, 1999. Google Scholar
  49. Haidong Shao, Min Xia, Jiafu Wan, and Clarence W de Silva. Modified stacked autoencoder using adaptive morlet wavelet for intelligent fault diagnosis of rotating machinery. IEEE/ASME Transactions on Mechatronics, 27(1):24-33, 2021. Google Scholar
  50. Nitish Srivastava and Russ R Salakhutdinov. Multimodal learning with deep boltzmann machines. Advances in neural information processing systems, 25, 2012. Google Scholar
  51. Yonas Tefera, Quinten Van Baelen, Maarten Meire, Stijn Luca, and Peter Karsmakers. Constraint-guided learning of data-driven health indicator models: An application on the pronostia bearing dataset. arXiv preprint arXiv:2503.09113, 2025. URL: https://doi.org/10.48550/arXiv.2503.09113.
  52. William Todo, Merwann Selmani, Béatrice Laurent, and Jean-Michel Loubes. Counterfactual explanation for multivariate times series using a contrastive variational autoencoder. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023. URL: https://doi.org/10.1109/ICASSP49357.2023.10095789.
  53. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. Google Scholar
  54. David C Westerly, Emilie Soisson, Quan Chen, Katherine Woch, Leah Schubert, Gustavo Olivera, and Thomas R Mackie. Treatment planning to improve delivery accuracy and patient throughput in helical tomotherapy. International Journal of Radiation Oncology* Biology* Physics, 74(4):1290-1297, 2009. Google Scholar
  55. Achmad Widodo, Toni Prahasto, Mochamad Soleh, and Herry Nugraha. Diagnostics and prognostics of boilers in power plant based on data-driven and machine learning. International Journal of Prognostics and Health Management, 16(1), 2025. Google Scholar
  56. Binbin Wu, Pengpeng Zhang, Bill Tsirakis, David Kanchaveli, and Thomas LoSasso. Utilizing historical mlc performance data from trajectory logs and service reports to establish a proactive maintenance model for minimizing treatment disruptions. Medical physics, 46(2):475-483, 2019. Google Scholar
  57. Tung-Yu Wu and You-Ting Wang. Locally interpretable one-class anomaly detection for credit card fraud detection. In 2021 International Conference on Technologies and Applications of Artificial Intelligence (TAAI), pages 25-30. IEEE, 2021. Google Scholar
  58. Yuting Wu, Mei Yuan, Shaopeng Dong, Li Lin, and Yingqi Liu. Remaining useful life estimation of engineered systems using vanilla lstm neural networks. Neurocomputing, 275:167-179, 2018. URL: https://doi.org/10.1016/J.NEUCOM.2017.05.063.
  59. Yiming Xu, Xiaohong Liu, Liyan Pan, Xiaojian Mao, Huiying Liang, Guangyu Wang, and Ting Chen. Explainable dynamic multimodal variational autoencoder for the prediction of patients with suspected central precocious puberty. IEEE Journal of Biomedical and Health Informatics, 26(3):1362-1373, 2021. URL: https://doi.org/10.1109/JBHI.2021.3103271.
  60. Jing Yang, Guo Xie, and Yanxi Yang. An improved ensemble fusion autoencoder model for fault diagnosis from imbalanced and incomplete data. Control Engineering Practice, 98:104358, 2020. Google Scholar
  61. Xitong Yang, Palghat Ramesh, Radha Chitta, Sriganesh Madhvanath, Edgar A Bernal, and Jiebo Luo. Deep multimodal representation learning from temporal data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5447-5455, 2017. Google Scholar
  62. Bang Xiang Yong and Alexandra Brintrup. Coalitional bayesian autoencoders: Towards explainable unsupervised deep learning with applications to condition monitoring under covariate shift. Applied Soft Computing, 123:108912, 2022. URL: https://doi.org/10.1016/J.ASOC.2022.108912.
  63. Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250, 2017. URL: https://arxiv.org/abs/1707.07250.
  64. Fei Zhao, Chengcui Zhang, and Baocheng Geng. Deep multimodal data fusion. ACM computing surveys, 56(9):1-36, 2024. URL: https://doi.org/10.1145/3649447.
  65. Zhiqian Zhao, Yeyin Xu, Jiabin Zhang, Runchao Zhao, Zhaobo Chen, and Yinghou Jiao. A semi-supervised gaussian mixture variational autoencoder method for few-shot fine-grained fault diagnosis. Neural Networks, 178:106482, 2024. URL: https://doi.org/10.1016/J.NEUNET.2024.106482.
  66. Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, and Michael R Lyu. Loghub: A large collection of system log datasets for ai-driven log analytics. In 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE), pages 355-366. IEEE, 2023. URL: https://doi.org/10.1109/ISSRE59848.2023.00071.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact