Document Open Access Logo

Tackling Domain-Specific Winograd Schemas with Knowledge-Based Reasoning and Machine Learning

Authors Suk Joon Hong, Brandon Bennett

PDF
Thumbnail PDF

File

OASIcs.LDK.2021.41.pdf
  • Filesize: 0.6 MB
  • 13 pages

Document Identifiers

Author Details

Suk Joon Hong
  • School of Mathematics, University of Leeds, UK
  • InfoMining Co., Seoul, South Korea
Brandon Bennett
  • School of Computing, University of Leeds, UK

Cite AsGet BibTex

Suk Joon Hong and Brandon Bennett. Tackling Domain-Specific Winograd Schemas with Knowledge-Based Reasoning and Machine Learning. In 3rd Conference on Language, Data and Knowledge (LDK 2021). Open Access Series in Informatics (OASIcs), Volume 93, pp. 41:1-41:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2021)
https://doi.org/10.4230/OASIcs.LDK.2021.41

Abstract

The Winograd Schema Challenge (WSC) is a commonsense reasoning task that requires background knowledge. In this paper, we contribute to tackling WSC in four ways. Firstly, we suggest a keyword method to define a restricted domain where distinctive high-level semantic patterns can be found. A thanking domain was defined by keywords, and the data set in this domain is used in our experiments. Secondly, we develop a high-level knowledge-based reasoning method using semantic roles which is based on the method of Sharma [Sharma, 2019]. Thirdly, we propose an ensemble method to combine knowledge-based reasoning and machine learning which shows the best performance in our experiments. As a machine learning method, we used Bidirectional Encoder Representations from Transformers (BERT) [Jacob Devlin et al., 2018; Vid Kocijan et al., 2019]. Lastly, in terms of evaluation, we suggest a "robust" accuracy measurement by modifying that of Trichelair et al. [Trichelair et al., 2018]. As with their switching method, we evaluate a model by considering its performance on trivial variants of each sentence in the test set.

Subject Classification

ACM Subject Classification
  • Computing methodologies → Artificial intelligence
Keywords
  • Commonsense Reasoning
  • Winograd Schema Challenge
  • Knowledge-based Reasoning
  • Machine Learning
  • Semantics

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Related Versions

Supplementary Materials

References

  1. Daniel Bailey, Amelia Harrison, Yuliya Lierler, Vladimir Lifschitz, and Julian Michael. The winograd schema challenge and reasoning about correlation. In 2015 AAAI Spring Symposium Series, USA, 2015. Google Scholar
  2. Brandon Bennett. Logical analysis of winograd schemas. Unpublished, 2020. Google Scholar
  3. Jacob Devlin, Ming W. Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv [cs.CL], 2018. URL: http://arxiv.org/abs/1810.04805.
  4. Allyson Ettinger. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48, 2020. Google Scholar
  5. Michael Gelfond and Vladimir Lifschitz. The stable model semantics for logic programming. In Proceedings of International Logic Programming Conference and Symposium, pages 1070-1080, 1988. Google Scholar
  6. Alfio Gliozzo and Carlo Strapparava. Semantic Domains in Computational Linguistics. Springer Berlin Heidelberg, 2009. Google Scholar
  7. Pascal Hitzler, Federico Bianchi, Monireh Ebrahimi, and Md Kamruzzaman Sarker. Neural-symbolic integration and the semantic web. Semantic Web, 11(1):3-11, 2020. Google Scholar
  8. Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD2004), 2004. Google Scholar
  9. Vid Kocijan, Ana M. Cretu, Oana M. Camburu, Yordan Yordanov, and Thomas Lukasiewicz. A surprisingly robust trick for winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837-–4842, 2019. Google Scholar
  10. Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In The 13th International Conference on Principles of Knowledge Representation and Reasoning, Italy, June 2012. Google Scholar
  11. Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari. Towards generalizable neuro-symbolic systems for commonsense question answering. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 22-32. Association for Computational Linguistics, 2019. Google Scholar
  12. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. Google Scholar
  13. Altaf Rahman and Vincent Ng. Resolving complex cases of definite pronouns: The winograd schema challenge. In EMNLP-CoNLL, 2012. Google Scholar
  14. Paul Rayson. From key words to key semantic domains. International Journal of Corpus Linguistics, 13(4):519-549, 2008. Google Scholar
  15. Melissa Roemmele, Cosmin A. Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning, USA, March 2011. Google Scholar
  16. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI-20, 2020. Google Scholar
  17. Arpit Sharma. Using answer set programming for commonsense reasoning in the winograd schema challenge. arXiv [cs.AI], 2019. URL: http://arxiv.org/abs/1907.11112.
  18. Arpit Sharma, Nguyen H. Vo, Somak Aditya, and Chitta Baral. Identifying various kinds of event mentions in k-parser output. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 82-88. Association for Computational Linguistics, 2015. Google Scholar
  19. Arpit Sharma, Nguyen H. Vo, Somak Aditya, and Chitta Baral. Towards addressing the winograd schema challenge - building and using a semantic parser and a knowledge hunting module. In IJCAI 2015, pages 1319-1325, 2015. Google Scholar
  20. Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, and Jackie C. K. Cheung. How reasonable are common-sense reasoning tasks: A case-study on the winograd schema challenge and swag. arXiv [cs.LG], 2018. URL: http://arxiv.org/abs/1811.01778.
  21. Trieu H. Trinh and Quoc V. Le. A simple method for commonsense reasoning. arXiv [cs.AI], 2018. URL: http://arxiv.org/abs/1806.02847.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact