Document Open Access Logo

A Neuro-Symbolic Approach to Structured Event Recognition

Authors Gianluca Apriceno, Andrea Passerini, Luciano Serafini

PDF
Thumbnail PDF

File

LIPIcs.TIME.2021.11.pdf
  • Filesize: 0.93 MB
  • 14 pages

Document Identifiers

Author Details

Gianluca Apriceno
  • University of Trento, Italy
  • Fondazione Bruno Kessler, Italy
Andrea Passerini
  • University of Trento, Italy
Luciano Serafini
  • Fondazione Bruno Kessler, Italy

Funding

This research was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.

Acknowledgements

The open access publication of this article was supported by the Alpen-Adria-Universität Klagenfurt, Austria.

Cite As Get BibTex

Gianluca Apriceno, Andrea Passerini, and Luciano Serafini. A Neuro-Symbolic Approach to Structured Event Recognition. In 28th International Symposium on Temporal Representation and Reasoning (TIME 2021). Leibniz International Proceedings in Informatics (LIPIcs), Volume 206, pp. 11:1-11:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2021) https://doi.org/10.4230/LIPIcs.TIME.2021.11

Abstract

Events are structured entities with multiple components: the event type, the participants with their roles, the outcome, the sub-events etc. A fully end-to-end approach for event recognition from raw data sequence, therefore, should also solve a number of simpler tasks like recognizing the objects involved in the events and their roles, the outcome of the events as well as the sub-events. Ontological knowledge about event structure, specified in logic languages, could be very useful to solve the aforementioned challenges. However, the majority of successful approaches in event recognition from raw data are based on purely neural approaches (mainly recurrent neural networks), with limited, if any, support for background knowledge. These approaches typically require large training sets with detailed annotations at the different levels in which recognition can be decomposed (e.g., video annotated with object bounding boxes, object roles, events and sub-events). In this paper, we propose a neuro-symbolic approach for structured event recognition from raw data that uses "shallow" annotation on the high-level events and exploits background knowledge to propagate this supervision to simpler tasks such as object classification. We develop a prototype of the approach and compare it with a purely neural solution based on recurrent neural networks, showing the higher capability of solving both the event recognition task and the simpler task of object classification, as well as the ability to generalize to events with unseen outcomes.

Subject Classification

ACM Subject Classification
  • Computing methodologies → Temporal reasoning
  • Computing methodologies → Activity recognition and understanding
Keywords
  • Event recognition
  • learning and reasoning
  • neuro-symbolic integration

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Kashif Ahmad Ahmad and Nicola Conci. How Deep Features Have Improved Event Recognition in Multimedia: A Survey. ACM Transactions on Multimedia Computing, Communications, and Applications, 15(2):39:1-39:27, 2019. URL: https://doi.org/10.1145/3306240.
  2. Elias Alevizos, Anastasios Skarlatidis, Alexander Artikis, and Georgios Paliouras. Probabilistic Complex Event Recognition: A Survey. ACM Computing Surveys, 50(5):71:1-71:31, 2017. URL: https://doi.org/10.1145/3117809.
  3. Alexander Artikis, Evangelos Makris, and Georgios Paliouras. A probabilistic interval-based event calculus for activity recognition. Annals of Mathematics and Artificial Intelligence, 89(1-2):29-52, 2021. URL: https://doi.org/10.1007/s10472-019-09664-4.
  4. Alexander Artikis, Anastasios Skarlatidis, François Portet, and Georgios Paliouras. Logic-based event recognition. The Knowledge Engineering Review, 27(4):469-506, 2012. URL: https://doi.org/10.1017/S0269888912000264.
  5. Caviar: context aware vision using image-based active recognition, 2011. URL: http://homepages.inf.ed.ac.uk/rbf/CAVIARDATA1.
  6. Liang-Chieh Chen, Alexander Schwing, Alan Yuille, and Raquel Urtasun. Learning Deep Structured Models. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1785-1794. JMLR.org, 2015. URL: http://proceedings.mlr.press/v37/chenb15.html.
  7. Kellie Corona, Katie Osterdahl, Roderic Collins, and Anthony Hoogs. MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detection. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pages 1059-1067. IEEE, 2021. URL: https://doi.org/10.1109/WACV48630.2021.00110.
  8. Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735-1780, 1997. URL: https://doi.org/10.1162/neco.1997.9.8.1735.
  9. Abdullah Khan, Loris Bozzato, Luciano Serafini, and Beatrice Lazzerini. Visual Reasoning on Complex Events in Soccer Videos Using Answer Set Programming. In Diego Calvanese and Luca Iocchi, editors, GCAI 2019. Proceedings of the 5th Global Conference on Artificial Intelligence, Bozen/Bolzano, Italy, 17-19 September 2019, volume 65, pages 42-53. EasyChair, 2019. URL: https://doi.org/10.29007/pjd4.
  10. Abdullah Khan, Luciano Serafini, Loris Bozzato, and Beatrice Lazzerini. Event Detection from Video Using Answer Set Programing. In Alberto Casagrande and Eugenio G. Omodeo, editors, Proceedings of the 34th Italian Conference on Computational Logic, Trieste, Italy, June 19-21, 2019, volume 2396 of CEUR Workshop Proceedings, pages 48-58. CEUR-WS.org, 2019. URL: http://ceur-ws.org/Vol-2396/paper25.pdf.
  11. Paula Lago, Shingo Takeda, Sayeda S Alia, Kohei Adachi, Brahim Bennai, and Sozo Inoue Francois Charpillet. A dataset for complex activity recognition withmicro and macro activities in a cooking scenario. CoRR, abs/2006.10681, 2020. URL: http://arxiv.org/abs/2006.10681.
  12. Weiyao Lin, Huabin Liu, Shizhan Liu, Yuxi Li, Rui Qian, Tao Wang, Ning Xu, Hongkai Xiong, Guo-Jun Qi, and Nicu Sebe. Human in Events: A Large-Scale Benchmark for Human-centric Video Analysis in Complex Events, 2020. URL: http://arxiv.org/abs/2005.04490.
  13. Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc D. Raedt. DeepProbLog: Neural Probabilistic Logic Programming. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, volume 31, pages 3753-3763, 2018. URL: https://proceedings.neurips.cc/paper/2018/hash/dc5d637ed5e62c36ecb73b654b05ba2a-Abstract.html.
  14. Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, and Marco Gori. LYRICS: A General Interface Layer to Integrate Logic Inference and deep Learning. In Ulf Brefeld, Élisa Fromont, Andreas Hotho, Arno J. Knobbe, Marloes H. Maathuis, and Céline Robardet, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II, volume 11907 of Lecture Notes in Computer Science, pages 283-298, 2019. URL: https://doi.org/10.1007/978-3-030-46147-8_17.
  15. Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, and Marco Gori. Integrating Learning and Reasoning with Deep Logic Models. In Ulf Brefeld, Élisa Fromont, Andreas Hotho, Arno J. Knobbe, Marloes H. Maathuis, and Céline Robardet, editors, Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part II, volume 11907 of Lecture Notes in Computer Science, pages 517-532. Springer, 2020. URL: https://doi.org/10.1007/978-3-030-46147-8_31.
  16. Juan C. Niebles, Chih-Wei Chen, and Li Fei-Fei. Modeling Temporal Structure of Decomposable Motion Segments for Activity Classification. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios, editors, Computer Vision - ECCV 2010, 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part II, volume 6312 of Lecture Notes in Computer Science, pages 392-405. Springer, 2010. URL: https://doi.org/10.1007/978-3-642-15552-9_29.
  17. Luc D. Raedt, Angelika Kimmig, and Hannu Toivonen. ProbLog: A Probabilistic Prolog and Its Application in Link Discovery. In Manuela M. Veloso, editor, IJCAI 2007, Proceedings of the 20th International Joint Conference on Artificial Intelligence, Hyderabad, India, January 6-12, 2007, pages 2462-2467, 2007. URL: http://ijcai.org/Proceedings/07/Papers/396.pdf.
  18. Fabrício Henrique Rodrigues and Mara Abel. What to consider about events: A survey on the ontology of occurrents. Applied Ontology, 14(4):343-378, 2019. URL: https://doi.org/10.3233/AO-190217.
  19. Luciano Serafini and Artur d'Avila Garcez. Learning and Reasoning with Logic Tensor Networks. In Giovanni Adorni, Stefano Cagnoni, Marco Gori, and Marco Maratea, editors, AI*IA 2016: Advances in Artificial Intelligence - XVth International Conference of the Italian Association for Artificial Intelligence, Genova, Italy, November 29 - December 1, 2016, Proceedings, volume 10037 of Lecture Notes in Computer Science, pages 334-348, 2016. URL: https://doi.org/10.1007/978-3-319-49130-1_25.
  20. Anastasios Skarlatidis, Alexander Artikis, Jason Filippou, and Georgios Paliouras. A probabilistic logic programming event calculus. Theory and Practice of Logic Programming, 15(2):213-245, 2015. URL: https://doi.org/10.1017/S1471068413000690.
  21. Khurram Soomro, Amir R. Zamir, and Mubarak Shah. UCF101: A dataset of 101 Human Actions Classes From Videos in The Wild. CoRR, abs/1212.0402, 2012. URL: http://arxiv.org/abs/1212.0402.
  22. Wei Xiang and Band Wan. A Survey of Event Extraction From Text. IEEE Access, 7:173111-173137, 2019. URL: https://doi.org/10.1109/ACCESS.2019.2956831.
  23. Tianwei Xing, Marc R. Vilamala, Luis Garcia, Federico Cerutti, Lance Kaplan, Alun Preece, and Mani Srivastava. DeepCEP: Deep Complex Event Processing Using Distributed Multimodal Information. In IEEE International Conference on Smart Computing, SMARTCOMP 2019, Washington, DC, USA, June 12-15, 2019, pages 87-92. IEEE, 2019. URL: https://doi.org/10.1109/SMARTCOMP.2019.00034.
  24. Zhun Yang, Adam Ishay, and Joohyung Lee. NeurASP: Embracing Neural Networks into Answer Set Programming. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 1755-1762. ijcai.org, 2020. URL: https://doi.org/10.24963/ijcai.2020/243.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact