LIPIcs.TIME.2022.12.pdf
- Filesize: 3.04 MB
- 19 pages
Document
A Neuro-Symbolic Approach for Real-World Event Recognition from Weak Supervision
Authors
-
Part of:
Volume:
29th International Symposium on Temporal Representation and Reasoning (TIME 2022)
Series: Leibniz International Proceedings in Informatics (LIPIcs)
Conference: International Symposium on Temporal Representation and Reasoning (TIME) - License:
Creative Commons Attribution 4.0 International license
- Publication Date: 2022-10-29
Cite AsGet BibTex
Gianluca Apriceno, Andrea Passerini, and Luciano Serafini. A Neuro-Symbolic Approach for Real-World Event Recognition from Weak Supervision. In 29th International Symposium on Temporal Representation and Reasoning (TIME 2022). Leibniz International Proceedings in Informatics (LIPIcs), Volume 247, pp. 12:1-12:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2022)
https://doi.org/10.4230/LIPIcs.TIME.2022.12
https://doi.org/10.4230/LIPIcs.TIME.2022.12
Abstract
Events are structured entities involving different components (e.g, the participants, their roles etc.) and their relations. Structured events are typically defined in terms of (a subset of) simpler, atomic events and a set of temporal relation between them. Temporal Event Detection (TED) is the task of detecting structured and atomic events within data streams, most often text or video sequences, and has numerous applications, from video surveillance to sports analytics. Existing deep learning approaches solve TED task by implicitly learning the temporal correlations among events from data. As consequence, these approaches often fail in ensuring a consistent prediction in terms of the relationship between structured and atomic events. On the other hand, neuro-symbolic approaches have shown their capability to constrain the output of the neural networks to be consistent with respect to the background knowledge of the domain. In this paper, we propose a neuro-symbolic approach for TED in a real world scenario involving sports activities. We show how by incorporating simple knowledge involving the relative order of atomic events and constraints on their duration, the approach substantially outperforms a fully neural solution in terms of recognition accuracy, when little or even no supervision is available on the atomic events.
Subject Classification
ACM Subject Classification
- Computing methodologies → Temporal reasoning
- Computing methodologies → Activity recognition and understanding
Keywords
- structured events
- temporal event detection
- neuro-symbolic integration
Metrics
- Access Statistics
-
Total Accesses (updated on a weekly basis)
0PDF Downloads
References
- Kashif Ahmad Ahmad and Nicola Conci. How Deep Features Have Improved Event Recognition in Multimedia: A Survey. ACM Transactions on Multimedia Computing, Communications, and Applications, 15(2):39:1-39:27, 2019. URL: https://doi.org/10.1145/3306240.
- Elias Alevizos, Anastasios Skarlatidis, Alexander Artikis, and Georgios Paliouras. Probabilistic Complex Event Recognition: A Survey. ACM Computing Surveys, 50(5):71:1-71:31, 2017. URL: https://doi.org/10.1145/3117809.
- Gianluca Apriceno, Andrea Passerini, and Luciano Serafini. A neuro-symbolic approach to structured event recognition. In Carlo Combi, Johann Eder, and Mark Reynolds, editors, 28th International Symposium on Temporal Representation and Reasoning, TIME 2021, September 27-29, 2021, Klagenfurt, Austria, volume 206 of LIPIcs, pages 11:1-11:14. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2021. URL: https://doi.org/10.4230/LIPIcs.TIME.2021.11.
- Alexander Artikis, Evangelos Makris, and Georgios Paliouras. A probabilistic interval-based event calculus for activity recognition. Annals of Mathematics and Artificial Intelligence, 89(1-2):29-52, 2021. URL: https://doi.org/10.1007/s10472-019-09664-4.
- Alexander Artikis, Anastasios Skarlatidis, François Portet, and Georgios Paliouras. Logic-based event recognition. The Knowledge Engineering Review, 27(4):469-506, 2012. URL: https://doi.org/10.1017/S0269888912000264.
- Samy Badreddine, Artur d'Avila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor networks. Artif. Intell., 303:103649, 2022. URL: https://doi.org/10.1016/j.artint.2021.103649.
- João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 4724-4733. IEEE Computer Society, 2017. URL: https://doi.org/10.1109/CVPR.2017.502.
- Ivan Donadello, Luciano Serafini, and Artur S. d'Avila Garcez. Logic tensor networks for semantic image interpretation. In Carles Sierra, editor, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 1596-1602. ijcai.org, 2017. URL: https://doi.org/10.24963/ijcai.2017/221.
- Paolo Dragone, Stefano Teso, and Andrea Passerini. Neuro-symbolic constraint programming for structured prediction. In Artur S. d'Avila Garcez and Ernesto Jiménez-Ruiz, editors, Proceedings of the 15th International Workshop on Neural-Symbolic Learning and Reasoning as part of the 1st International Joint Conference on Learning & Reasoning (IJCLR 2021), Virtual conference, October 25-27, 2021, volume 2986 of CEUR Workshop Proceedings, pages 6-14. CEUR-WS.org, 2021. URL: http://ceur-ws.org/Vol-2986/paper2.pdf.
- José Roldán Gómez, Juan Boubeta-Puig, José Luis Martínez, and Guadalupe Ortiz. Integrating complex event processing and machine learning: An intelligent architecture for detecting iot security attacks. Expert Syst. Appl., 149:113251, 2020. URL: https://doi.org/10.1016/j.eswa.2020.113251.
-
Pascal Hitzler and Md Kamruzzaman Sarker. Neuro-Symbolic Artificial Intelligence: The State of the Art. IOS Press, 2022.
- Jeya Vikranth Jeyakumar, Luke Dickens, Luis Garcia, Yu-Hsi Cheng, Diego Ramirez-Echavarria, Joseph Noor, Alessandra Russo, Lance M. Kaplan, Erik Blasch, and Mani B. Srivastava. Automatic concept extraction for concept bottleneck-based video classification. CoRR, abs/2206.10129, 2022. URL: https://doi.org/10.48550/arXiv.2206.10129.
- Abdullah Khan, Loris Bozzato, Luciano Serafini, and Beatrice Lazzerini. Visual Reasoning on Complex Events in Soccer Videos Using Answer Set Programming. In Diego Calvanese and Luca Iocchi, editors, GCAI 2019. Proceedings of the 5th Global Conference on Artificial Intelligence, Bozen/Bolzano, Italy, 17-19 September 2019, volume 65, pages 42-53. EasyChair, 2019. URL: https://doi.org/10.29007/pjd4.
- Abdullah Khan, Luciano Serafini, Loris Bozzato, and Beatrice Lazzerini. Event Detection from Video Using Answer Set Programing. In Alberto Casagrande and Eugenio G. Omodeo, editors, Proceedings of the 34th Italian Conference on Computational Logic, Trieste, Italy, June 19-21, 2019, volume 2396 of CEUR Workshop Proceedings, pages 48-58. CEUR-WS.org, 2019. URL: http://ceur-ws.org/Vol-2396/paper25.pdf.
- Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5338-5348. PMLR, 2020. URL: http://proceedings.mlr.press/v119/koh20a.html.
- Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc D. Raedt. DeepProbLog: Neural Probabilistic Logic Programming. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, volume 31, pages 3753-3763, 2018. URL: https://proceedings.neurips.cc/paper/2018/hash/dc5d637ed5e62c36ecb73b654b05ba2a-Abstract.html.
- Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, and Marco Gori. LYRICS: A General Interface Layer to Integrate Logic Inference and deep Learning. In Ulf Brefeld, Élisa Fromont, Andreas Hotho, Arno J. Knobbe, Marloes H. Maathuis, and Céline Robardet, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II, volume 11907 of Lecture Notes in Computer Science, pages 283-298, 2019. URL: https://doi.org/10.1007/978-3-030-46147-8_17.
- Luc D. Raedt, Angelika Kimmig, and Hannu Toivonen. ProbLog: A Probabilistic Prolog and Its Application in Link Discovery. In Manuela M. Veloso, editor, IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6-12, 2007, pages 2462-2467, 2007. URL: http://ijcai.org/Proceedings/07/Papers/396.pdf.
-
Roberto Sebastiani and Silvia Tomasi. Optimization Modulo Theories with Linear Rational Costs. ACM Transactions on Computational Logics, 16(2), 2015.
- Anastasios Skarlatidis, Alexander Artikis, Jason Filippou, and Georgios Paliouras. A probabilistic logic programming event calculus. Theory and Practice of Logic Programming, 15(2):213-245, 2015. URL: https://doi.org/10.1017/S1471068413000690.
- Praveen Tirupattur, Kevin Duarte, Yogesh Singh Rawat, and Mubarak Shah. Modeling multi-label action dependencies for temporal action localization. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 1460-1470. Computer Vision Foundation / IEEE, 2021. URL: https://openaccess.thecvf.com/content/CVPR2021/html/Tirupattur_Modeling_Multi-Label_Action_Dependencies_for_Temporal_Action_Localization_CVPR_2021_paper.html, URL: https://doi.org/10.1109/CVPR46437.2021.00151.
- Marc Roig Vilamala, Liam Hiley, Yulia Hicks, Alun D. Preece, and Federico Cerutti. A pilot study on detecting violence in videos fusing proxy models. In 22th International Conference on Information Fusion, FUSION 2019, Ottawa, ON, Canada, July 2-5, 2019, pages 1-8. IEEE, 2019. URL: https://ieeexplore.ieee.org/document/9011329.
- Marc Roig Vilamala, Tianwei Xing, Harrison Taylor, Luis Garcia, Mani B. Srivastava, Lance M. Kaplan, Alun D. Preece, Angelika Kimmig, and Federico Cerutti. Using deepproblog to perform complex event processing on an audio stream. CoRR, abs/2110.08090, 2021. URL: http://arxiv.org/abs/2110.08090.
- Thomas Winters, Giuseppe Marra, Robin Manhaeve, and Luc De Raedt. Deepstochlog: Neural stochastic logic programming. CoRR, abs/2106.12574, 2021. URL: http://arxiv.org/abs/2106.12574.
- Wei Xiang and Band Wan. A Survey of Event Extraction From Text. IEEE Access, 7:173111-173137, 2019. URL: https://doi.org/10.1109/ACCESS.2019.2956831.
- Tianwei Xing, Luis Garcia, Marc Roig Vilamala, Federico Cerutti, Lance M. Kaplan, Alun D. Preece, and Mani B. Srivastava. Neuroplex: learning to detect complex events in sensor networks through knowledge injection. In Jin Nakazawa and Polly Huang, editors, SenSys '20: The 18th ACM Conference on Embedded Networked Sensor Systems, Virtual Event, Japan, November 16-19, 2020, pages 489-502. ACM, 2020. URL: https://doi.org/10.1145/3384419.3431158.
- Tianwei Xing, Marc R. Vilamala, Luis Garcia, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. DeepCEP: Deep Complex Event Processing Using Distributed Multimodal Information. In IEEE International Conference on Smart Computing, SMARTCOMP 2019, Washington, DC, USA, June 12-15, 2019, pages 87-92. IEEE, 2019. URL: https://doi.org/10.1109/SMARTCOMP.2019.00034.
- Zhun Yang, Adam Ishay, and Joohyung Lee. NeurASP: Embracing Neural Networks into Answer Set Programming. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 1755-1762. ijcai.org, 2020. URL: https://doi.org/10.24963/ijcai.2020/243.
- Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, and Li Fei-Fei. Every moment counts: Dense detailed labeling of actions in complex videos. Int. J. Comput. Vis., 126(2-4):375-389, 2018. URL: https://doi.org/10.1007/s11263-017-1013-y.