Document Open Access Logo

RTScale: Sensitivity-Aware Adaptive Image Scaling for Real-Time Object Detection

Authors Seonyeong Heo , Shinnung Jeong, Hanjun Kim

PDF
Thumbnail PDF

File

LIPIcs.ECRTS.2022.2.pdf
  • Filesize: 1.64 MB
  • 22 pages

Document Identifiers

Author Details

Seonyeong Heo
  • Department of Information Technology and Electrical Engineering, ETH Zürich, Switzerland
Shinnung Jeong
  • Department of Electrical and Electronic Engineering, Yonsei University, Seoul, Republic of Korea
Hanjun Kim
  • Department of Electrical and Electronic Engineering, Yonsei University, Seoul, Republic of Korea

Funding

This work is supported by IITP-2020-0-01847, IITP-2020-0-01361, and IITP-2021-0-00853 through the Institute of Information and Communication Technology Planning and Evaluation (IITP) funded by the Ministry of Science and ICT. This work is also supported by Samsung Electronics.

Acknowledgements

We thank the anonymous reviewers for their valuable feedback. We also thank the CoreLab members for their support and feedback during this work. (Corresponding author: Hanjun Kim)

Cite AsGet BibTex

Seonyeong Heo, Shinnung Jeong, and Hanjun Kim. RTScale: Sensitivity-Aware Adaptive Image Scaling for Real-Time Object Detection. In 34th Euromicro Conference on Real-Time Systems (ECRTS 2022). Leibniz International Proceedings in Informatics (LIPIcs), Volume 231, pp. 2:1-2:22, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2022)
https://doi.org/10.4230/LIPIcs.ECRTS.2022.2

Abstract

Real-time object detection is crucial in autonomous driving. To avoid catastrophic accidents, an autonomous car should detect objects with multiple cameras and make decisions within a certain time limit. Object detection systems can meet the real-time constraint by dynamically downsampling input images to proper scales according to their time budget. However, simply applying the same scale to all the images from multiple cameras can cause unnecessary accuracy loss because downsampling can incur a significant accuracy loss for some images. To reduce the accuracy loss while meeting the real-time constraint, this work proposes RTScale, a new adaptive real-time image scaling scheme that applies different scales to different images reflecting their sensitivities to the scaling and time budget. RTScale infers the sensitivities of multiple images from multiple cameras and determines an appropriate image scale for each image considering the real-time constraint. This work evaluates object detection accuracy and latency with RTScale for two driving datasets. The evaluation results show that RTScale can meet real-time constraints with minimal accuracy loss.

Subject Classification

ACM Subject Classification
  • Computer systems organization → Real-time systems
  • Computer systems organization → Parallel architectures
  • Software and its engineering → Real-time systems software
  • Computing methodologies → Neural networks
  • Computing methodologies → Object detection
  • Theory of computation → Scheduling algorithms
Keywords
  • Real-time object detection
  • Dynamic neural network execution
  • Adaptive image scaling
  • Autonomous driving
  • Self-driving cars

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Supplementary Materials

References

  1. Apollo. URL: https://apollo.auto/index.html.
  2. Autoware.ai Core Perception Github Repository. https://github.com/Autoware-AI/core_perception, May 2021.
  3. Soroush Bateni and Cong Liu. ApNet: Approximation-aware real-time neural network. In 2018 IEEE Real-Time Systems Symposium (RTSS), 2018. URL: https://doi.org/10.1109/RTSS.2018.00017.
  4. Soroush Bateni, Husheng Zhou, Yuankun Zhu, and Cong Liu. PredJoule: A timing-predictable energy optimization framework for deep neural networks. In 2018 IEEE Real-Time Systems Symposium (RTSS), December 2018. URL: https://doi.org/10.1109/RTSS.2018.00020.
  5. Adam Betts and Alastair Donaldson. Estimating the wcet of gpu-accelerated applications using hybrid analysis. In 2013 25th Euromicro Conference on Real-Time Systems (ECRTS), 2013. URL: https://doi.org/10.1109/ECRTS.2013.29.
  6. Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. YOLOv4: Optimal speed and accuracy of object detection, 2020. URL: https://doi.org/10.48550/arXiv.2004.10934.
  7. Ting-Wu Chin, Ruizhou Ding, and Diana Marculescu. AdaScale: Towards real-time video object detection using adaptive scaling. In Proceedings of Machine Learning and Systems 2019, pages 431-441. 2019. Google Scholar
  8. Ting-Wu Chin, Chia-Lin Yu, Matthew Halpern, Hasan Genc, Shiao-Li Tsao, and Vijay Janapa Reddi. Domain-specific approximation for object detection. IEEE Micro, 38(1):31-40, 2018. URL: https://doi.org/10.1109/MM.2018.112130335.
  9. Hiroyuki Chishiro, Kazutoshi Suito, Tsutomu Ito, Seiya Maeda, Takuya Azumi, Kenji Funaoka, and Shinpei Kato. Towards heterogeneous computing platforms for autonomous driving. In 2019 IEEE International Conference on Embedded Software and Systems (ICESS), 2019. URL: https://doi.org/10.1109/ICESS.2019.8782446.
  10. Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Towards the limit of network quantization. In Proceedings of the 5th International Conference on Learning Representations, ICLR ’17, 2017. Google Scholar
  11. Darknet. URL: https://github.com/AlexeyAB/darknet.
  12. Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012. URL: https://doi.org/10.1109/CVPR.2012.6248074.
  13. Ross Girshick. Fast R-CNN. 2015 IEEE International Conference on Computer Vision (ICCV), 2015. URL: https://doi.org/10.1109/ICCV.2015.169.
  14. Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014. URL: https://doi.org/10.1109/CVPR.2014.81.
  15. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15. JMLR.org, 2015. Google Scholar
  16. Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, Cambridge, MA, USA, 2015. MIT Press. Google Scholar
  17. Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV), October 2017. URL: https://doi.org/10.1109/ICCV.2017.322.
  18. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. URL: https://doi.org/10.1109/CVPR.2016.90.
  19. Seonyeong Heo, Sungjun Cho, Youngsok Kim, and Hanjun Kim. Real-time object detection system with multi-path neural networks. In 2020 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2020. URL: https://doi.org/10.1109/RTAS48715.2020.000-8.
  20. Vesa Hirvisalo. On static timing analysis of gpu kernels. In 14th International Workshop on Worst-Case Execution Time Analysis, OpenAccess Series in Informatics (OASIcs), 2014. Google Scholar
  21. Yijie Huangfu and Wei Zhang. Static wcet analysis of gpus with predictable warp scheduling. In 2017 IEEE International Symposium on Real-Time Computing (ISORC), 2017. URL: https://doi.org/10.1109/ISORC.2017.24.
  22. Wonseok Jang, Hansaem Jeong, Kyungtae Kang, Nikil Dutt, and Jong-Chan Kim. R-TOD: Real-time object detector with minimized end-to-end delay for autonomous driving. In 2020 IEEE Real-Time Systems Symposium (RTSS), pages 191-204, 2020. URL: https://doi.org/10.1109/RTSS49844.2020.00027.
  23. Woochul Kang and Jaeyong Chung. DeepRT: Predictable deep learning inference for cyber-physical systems. Real-Time Systems, 55(1):106-135, January 2019. URL: https://doi.org/10.1007/s11241-018-9314-y.
  24. Shinpei Kato, Eijiro Takeuchi, Yoshio Ishiguro, Yoshiki Ninomiya, Kazuya Takeda, and Tsuyoshi Hamada. An open approach to autonomous vehicles. IEEE Micro, 35(6):60-68, 2015. URL: https://doi.org/10.1109/MM.2015.133.
  25. Shinpei Kato, Shota Tokunaga, Yuya Maruyama, Seiya Maeda, Manato Hirabayashi, Yuki Kitsukawa, Abraham Monrroy, Tomohito Ando, Yusuke Fujii, and Takuya Azumi. Autoware on board: Enabling autonomous vehicles with embedded systems. In Proceedings of the 9th ACM/IEEE International Conference on Cyber-Physical Systems, ICCPS '18, pages 287-296. IEEE Press, 2018. URL: https://doi.org/10.1109/ICCPS.2018.00035.
  26. Jung-Eun Kim, Richard Bradford, Man-Ki Yoon, and Zhong Shao. ABC: Abstract prediction before concreteness. In 2020 Design, Automation Test in Europe Conference Exhibition (DATE), pages 1103-1108, 2020. URL: https://doi.org/10.23919/DATE48585.2020.9116479.
  27. Seulki Lee and Shahriar Nirjon. SubFlow: A dynamic induced-subgraph strategy toward real-time dnn inference and training. In 2020 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2020. URL: https://doi.org/10.1109/RTAS48715.2020.00-20.
  28. Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In Proceedings of the 5th International Conference on Learning Representations, ICLR ’17, 2017. Google Scholar
  29. Shih-Chieh Lin, Yunqi Zhang, Chang-Hong Hsu, Matt Skach, Md E. Haque, Lingjia Tang, and Jason Mars. The architectural implications of autonomous driving: Constraints and acceleration. In Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '18. Association for Computing Machinery, 2018. URL: https://doi.org/10.1145/3173162.3173191.
  30. Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. URL: https://doi.org/10.1109/CVPR.2017.106.
  31. Shengzhong Liu, Shuochao Yao, Xinzhe Fu, Huajie Shao, Rohan Tabish, Simon Yu, Ayoosh Bansal, Heechul Yun, Lui Sha, and Tarek Abdelzaher. Real-time task scheduling for machine perception in in intelligent cyber-physical systems. IEEE Transactions on Computers, 2021. URL: https://doi.org/10.1109/TC.2021.3106496.
  32. Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision - ECCV 2016. Springer International Publishing, 2016. URL: https://doi.org/10.1007/978-3-319-46448-0_2.
  33. Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In Proceedings of the 30th International Conference on Machine Learning, 2013. Google Scholar
  34. Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 779-788, 2016. URL: https://doi.org/10.1109/CVPR.2016.91.
  35. Joseph Redmon and Ali Farhadi. YOLO9000: Better, faster, stronger. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. URL: https://doi.org/10.1109/CVPR.2017.690.
  36. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 2017. URL: https://doi.org/10.1109/TPAMI.2016.2577031.
  37. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. URL: https://doi.org/10.1109/CVPR.2015.7298594.
  38. Hamid Tabani, Matteo Fusi, Leonidas Kosmidis, Jaume Abella, and Francisco J. Cazorla. IntPred: Flexible, fast, and accurate object detection for autonomous driving systems. In Proceedings of the 35th Annual ACM Symposium on Applied Computing, SAC ’20. Association for Computing Machinery, 2020. URL: https://doi.org/10.1145/3341105.3373918.
  39. Tesla Model S Owners Manual. https://www.tesla.com/sites/default/files/model_s_owners_manual_north_america_en_us.pdf, April 2020.
  40. Vijay V. Vazirani. Approximation Algorithms. Springer Publishing Company, Incorporated, 2010. URL: https://doi.org/10.1007/978-3-662-04565-7.
  41. Yecheng Xiang and Hyoseung Kim. Pipelined data-parallel cpu/gpu scheduling for multi-dnn real-time inference. In 2019 IEEE Real-Time Systems Symposium (RTSS), pages 392-405, 2019. URL: https://doi.org/10.1109/RTSS46320.2019.00042.
  42. Shuochao Yao, Yifan Hao, Yiran Zhao, Huajie Shao, Dongxin Liu, Shengzhong Liu, Tianshi Wang, Jinyang Li, and Tarek Abdelzaher. Scheduling real-time deep learning services as imprecise computations. In 2020 IEEE 26th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), pages 1-10, 2020. URL: https://doi.org/10.1109/RTCSA50079.2020.9203676.
  43. Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. URL: https://doi.org/10.1109/CVPR42600.2020.00271.
  44. Xiaofan Zhang, Haoming Lu, Cong Hao, Jiachen Li, Bowen Cheng, Yuhong Li, Kyle Rupnow, Jinjun Xiong, Thomas Huang, Honghui Shi, Wen-Mei Hwu, and Deming Chen. SkyNet: a hardware-efficient method for object detection and tracking on embedded systems. In Proceedings of Machine Learning and Systems 2020, pages 216-229. 2020. Google Scholar
  45. Husheng Zhou, Soroush Bateni, and Cong Liu. S³DNN: Supervised streaming and scheduling for gpu-accelerated real-time dnn workloads. In 2018 IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), pages 190-201, 2018. URL: https://doi.org/10.1109/RTAS.2018.00028.
  46. Menglong Zhu and Mason Liu. Mobile video object detection with temporally-aware feature maps. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5686-5695, 2018. URL: https://doi.org/10.1109/CVPR.2018.00596.
  47. Xizhou Zhu, Jifeng Dai, Lu Yuan, and Yichen Wei. Towards high performance video object detection. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7218, 2018. URL: https://doi.org/10.1109/CVPR.2018.00753.
  48. Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 408-417, 2017. URL: https://doi.org/10.1109/ICCV.2017.52.
  49. Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei. Deep feature flow for video recognition. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4141-4150, 2017. URL: https://doi.org/10.1109/CVPR.2017.441.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact