Document Open Access Logo

Alignment Sets for Sensor Fusion Against Temporal Misalignment

Authors Daniel Kuhse , Mario Günzel , Harun Teper , Lars Willemsen , Georg von der Brüggen , Jian-Jia Chen

PDF

File

Thumbnail PDF
PDF
LIPIcs.ECRTS.2026.8.pdf
  • Filesize: 2.34 MB
  • 24 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computer systems organization → Real-time system specification
  • Computer systems organization → Embedded software
Keywords
  • Sensor Fusion
  • Temporal Misalignment
  • Robustness
  • Timing Analysis

Metrics

Abstract

Sensor fusion algorithms combine data from multiple sensors to produce more accurate and reliable results. However, temporal misalignment between sensors, caused by factors such as clock drift, jitter or networking delays, can significantly degrade fusion quality. Prior work on modeling temporal misalignment in sensor fusion algorithms assumes that in the ideal case all samples should be aligned with the same reference time point. We show that this assumption limits its applicability when samples are intentionally taken at different time points, e.g., when a single sensor is sampled multiple times or when sensors operate at different frequencies.
In this paper, we introduce alignment sets, which allow system designers to explicitly specify the intended alignment between samples. This flexibility enables more precise temporal misalignment measures that better reflect the actual requirements of sensor fusion scenarios. We prove that alignment sets generalize the prior definitions of temporal misalignment of sensor fusion algorithms. We also provide an evaluation on a camera-LiDAR fusion pipeline for 3D object detection, showing that alignment sets provide more accurate misalignment measures and robustness estimates.

Cite As Get BibTex

Daniel Kuhse, Mario Günzel, Harun Teper, Lars Willemsen, Georg von der Brüggen, and Jian-Jia Chen. Alignment Sets for Sensor Fusion Against Temporal Misalignment. In 38th European Conference on Real-Time Systems (ECRTS 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 375, pp. 8:1-8:24, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ECRTS.2026.8

Author Details

Daniel Kuhse
  • TU Dortmund, Germany
Mario Günzel
  • TU Dortmund, Germany
Harun Teper
  • TU Dortmund, Germany
Lars Willemsen
  • TU Dortmund, Germany
Georg von der Brüggen
  • TU Dortmund, Germany
Jian-Jia Chen
  • RWTH Aachen University, Germany
  • TU Dortmund, Germany

Funding

This work has received funding by the German Federal Ministry of Education and Research (BMBF) in the course of the 6GEM research hub under grant number 16KISK038. This result is part of a project (PropRT) that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 865170). This work has been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - Project No. 569077889 (PEACH). This work has been funded by the Federal Ministry of Research, Technology and Space (BMFTR) via the 6GEM+ transfer hub under funding references 16KIS2412.

References

  1. Michael Aeberhard and Nico Kaempchen. High-level sensor data fusion architecture for vehicle surround environment perception. In Proc. 8th Int. Workshop Intell. Transp, volume 665, 2011. Google Scholar
  2. Eugene Asarin, Alexandre Donzé, Oded Maler, and Dejan Nickovic. Parametric identification of temporal properties. In Sarfraz Khurshid and Koushik Sen, editors, Runtime Verification - Second International Conference, RV 2011, San Francisco, CA, USA, September 27-30, 2011, Revised Selected Papers, Lecture Notes in Computer Science, pages 147-160. Springer, 2011. URL: https://doi.org/10.1007/978-3-642-29860-8_12.
  3. Michal Bednarek, Piotr Kicki, and Krzysztof Walas. On robustness of multi-modal fusion—robotics perspective. Electronics, 9(7):1152, 2020. Google Scholar
  4. Said Yacine Boulahia, Abdenour Amamra, Mohamed Ridha Madi, and Said Daikh. Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition. Machine Vision and Applications, 32(6):121, 2021. URL: https://doi.org/10.1007/s00138-021-01249-8.
  5. Varuna De Silva, Jamie Roche, and Ahmet Kondoz. Robust fusion of lidar and wide-angle camera data for autonomous mobile robots. Sensors, 18(8):2730, 2018. URL: https://doi.org/10.3390/s18082730.
  6. Alexandre Donzé. Breach, A toolbox for verification and parameter synthesis of hybrid systems. In Tayssir Touili, Byron Cook, and Paul B. Jackson, editors, Computer Aided Verification, 22nd International Conference, CAV 2010, Edinburgh, UK, July 15-19, 2010. Proceedings, Lecture Notes in Computer Science, pages 167-170. Springer, 2010. URL: https://doi.org/10.1007/978-3-642-14295-6_17.
  7. Alexandre Donzé, Thomas Ferrère, and Oded Maler. Efficient robust monitoring for STL. In Natasha Sharygina and Helmut Veith, editors, Computer Aided Verification - 25th International Conference, CAV 2013, Saint Petersburg, Russia, July 13-19, 2013. Proceedings, Lecture Notes in Computer Science, pages 264-279. Springer, 2013. URL: https://doi.org/10.1007/978-3-642-39799-8_19.
  8. Alexandre Donzé and Oded Maler. Robust satisfaction of temporal logic over real-valued signals. In Krishnendu Chatterjee and Thomas A. Henzinger, editors, Formal Modeling and Analysis of Timed Systems - 8th International Conference, FORMATS 2010, Klosterneuburg, Austria, September 8-10, 2010. Proceedings, Lecture Notes in Computer Science, pages 92-106. Springer, 2010. URL: https://doi.org/10.1007/978-3-642-15297-9_9.
  9. Georgios E. Fainekos and George J. Pappas. Robustness of temporal logic specifications for continuous-time signals. Theor. Comput. Sci., 410(42):4262-4291, 2009. URL: https://doi.org/10.1016/j.tcs.2009.06.021.
  10. Thomas Ferrere. Assertions and measurements for mixed-signal simulation. Ph. D. Dissertation, 2016. Google Scholar
  11. Andreas Finkenzeller, Andrew Roberts, Mauro Bellone, Olaf Maennel, Mohammad Hamad, and Sebastian Steinhorst. Sensor fusion desynchronization attacks. In 37th Euromicro Conference on Real-Time Systems (ECRTS 2025), pages 6:1-6:22. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2025. URL: https://doi.org/10.4230/LIPIcs.ECRTS.2025.6.
  12. Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231-1237, 2013. URL: https://doi.org/10.1177/0278364913491297.
  13. Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV, Lecture Notes in Computer Science, pages 624-642. Springer, 2022. URL: https://doi.org/10.1007/978-3-031-19781-9_36.
  14. Shinpei Kato, Shota Tokunaga, Yuya Maruyama, Seiya Maeda, Manato Hirabayashi, Yuki Kitsukawa, Abraham Monrroy, Tomohito Ando, Yusuke Fujii, and Takuya Azumi. Autoware on board: Enabling autonomous vehicles with embedded systems. In 2018 ACM/IEEE 9th International Conference on Cyber-Physical Systems (ICCPS), pages 287-296. IEEE, 2018. URL: https://doi.org/10.1109/ICCPS.2018.00035.
  15. Taewan Kim and Joydeep Ghosh. On single source robustness in deep fusion models. Advances in Neural Information Processing Systems, 32, 2019. Google Scholar
  16. Daniel Kuhse, Nils Hölscher, Mario Günzel, Harun Teper, Georg von der Brüggen, Jian-Jia Chen, and Ching-Chi Lin. Sync or sink? the robustness of sensor fusion against temporal misalignment. In 30th IEEE Real-Time and Embedded Technology and Applications Symposium, RTAS 2024, Hong Kong, May 13-16, 2024, pages 122-134. IEEE, 2024. URL: https://doi.org/10.1109/RTAS61025.2024.00018.
  17. Elmar Mair, Michael Fleps, Michael Suppa, and Darius Burschka. Spatio-temporal initialization for imu to camera registration. In 2011 IEEE International Conference on Robotics and Biomimetics, pages 557-564. IEEE, 2011. URL: https://doi.org/10.1109/ROBIO.2011.6181345.
  18. Oded Maler and Dejan Nickovic. Monitoring temporal properties of continuous signals. In Yassine Lakhnech and Sergio Yovine, editors, Formal Techniques, Modelling and Analysis of Timed and Fault-Tolerant Systems, Joint International Conferences on Formal Modelling and Analysis of Timed Systems, FORMATS 2004 and Formal Techniques in Real-Time and Fault-Tolerant Systems, FTRTFT 2004, Grenoble, France, September 22-24, 2004, Proceedings, Lecture Notes in Computer Science, pages 152-166. Springer, 2004. URL: https://doi.org/10.1007/978-3-540-30206-3_12.
  19. Dejan Nickovic and Tomoya Yamaguchi. RTAMT: online robustness monitors from STL. In Dang Van Hung and Oleg Sokolsky, editors, Automated Technology for Verification and Analysis - 18th International Symposium, ATVA 2020, Hanoi, Vietnam, October 19-23, 2020, Proceedings, Lecture Notes in Computer Science, pages 564-571. Springer, 2020. URL: https://doi.org/10.1007/978-3-030-59152-6_34.
  20. Chanoh Park, Peyman Moghadam, Soohwan Kim, Sridha Sridharan, and Clinton Fookes. Spatiotemporal camera-lidar calibration: A targetless and structureless approach. IEEE Robotics and Automation Letters, 5(2):1556-1563, 2020. URL: https://doi.org/10.1109/LRA.2020.2969164.
  21. Won Park, Nan Liu, Qi Alfred Chen, and Z Morley Mao. Sensor adversarial traits: Analyzing robustness of 3d object detection sensor fusion models. In 2021 IEEE International Conference on Image Processing (ICIP), pages 484-488. IEEE, 2021. URL: https://doi.org/10.1109/ICIP42928.2021.9506183.
  22. Tong Qin and Shaojie Shen. Online temporal calibration for monocular visual-inertial systems. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3662-3669. IEEE, 2018. URL: https://doi.org/10.1109/IROS.2018.8593603.
  23. Neha Rino, Mohammed Foughali, and Eugene Asarin. Efficiently computable distance-based robustness for a practical fragment of STL. In Jane Hillston, Sadegh Soudjani, and Masaki Waga, editors, Quantitative Evaluation of Systems and Formal Modeling and Analysis of Timed Systems - First International Joint Conference, QEST+FORMATS 2024, Calgary, AB, Canada, September 9-13, 2024, Proceedings, Lecture Notes in Computer Science, pages 179-195. Springer, 2024. URL: https://doi.org/10.1007/978-3-031-68416-6_11.
  24. Neha Rino, Mohammed Foughali, Florian Renkin, and Eugene Asarin. Efficiently computable temporal robustness for a practical stl fragment. International Journal on Software Tools for Technology Transfer, March 2026. URL: https://doi.org/10.1007/s10009-026-00852-2.
  25. Md Hasan Shahriar, Md Mohaimin Al Barat, Harshavardhan Sundar, Ning Zhang, Naren Ramakrishnan, Y Thomas Hou, and Wenjing Lou. Temporal misalignment attacks against multimodal perception in autonomous driving. arXiv preprint, 2025. URL: https://arxiv.org/abs/2507.09095.
  26. Yde Sinnema and Martina Maggio. Analysis of control systems under sensor timing misalignments. In 2025 IEEE 31st Real-Time and Embedded Technology and Applications Symposium (RTAS), pages 137-150. IEEE, 2025. URL: https://doi.org/10.1109/RTAS65571.2025.00024.
  27. James Tu, Huichen Li, Xinchen Yan, Mengye Ren, Yun Chen, Ming Liang, Eilyan Bitar, Ersin Yumer, and Raquel Urtasun. Exploring adversarial robustness of multi-sensor perception systems in self driving. arXiv preprint, 2021. URL: https://arxiv.org/abs/2101.06784.
  28. Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao. Yolov9: Learning what you want to learn using programmable gradient information. In European conference on computer vision, pages 1-21. Springer, 2024. URL: https://doi.org/10.1007/978-3-031-72751-1_1.
  29. Shaojie Wang, Tong Wu, Ayan Chakrabarti, and Yevgeniy Vorobeychik. Adversarial robustness of deep sensor fusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2387-2396, 2022. Google Scholar
  30. Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer Publishing Company, Incorporated, 2010. Google Scholar
  31. Antje Westenberger, Tobias Huck, Martin Fritzsche, Tilo Schwarz, and Klaus Dietmayer. Temporal synchronization in multi-sensor fusion for future driver assistance systems. In 2011 IEEE International Symposium on Precision Clock Synchronization for Measurement, Control and Communication, pages 93-98. IEEE, 2011. Google Scholar
  32. Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784-11793, 2021. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact