Document Open Access Logo

Explainable Zero-Shot Topic Extraction Using a Common-Sense Knowledge Graph

Authors Ismail Harrando , Raphaël Troncy

PDF
Thumbnail PDF

File

OASIcs.LDK.2021.17.pdf
  • Filesize: 0.95 MB
  • 15 pages

Document Identifiers

Author Details

Ismail Harrando
  • EURECOM, Sophia Antipolis, Biot, France
Raphaël Troncy
  • EURECOM, Sophia Antipolis, Biot, France

Funding

This work has been partially supported by the French National Research Agency (ANR) within the ASRAEL (ANR-15-CE23-0018) and ANTRACT (ANR-17-CE38-0010) projects, and by the European Union’s Horizon 2020 research and innovation program within the MeMAD (GA 780069) and SILKNOW (GA 769504) projects.

Cite As Get BibTex

Ismail Harrando and Raphaël Troncy. Explainable Zero-Shot Topic Extraction Using a Common-Sense Knowledge Graph. In 3rd Conference on Language, Data and Knowledge (LDK 2021). Open Access Series in Informatics (OASIcs), Volume 93, pp. 17:1-17:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2021) https://doi.org/10.4230/OASIcs.LDK.2021.17

Abstract

Pre-trained word embeddings constitute an essential building block for many NLP systems and applications, notably when labeled data is scarce. However, since they compress word meanings into a fixed-dimensional representation, their use usually lack interpretability beyond a measure of similarity and linear analogies that do not always reflect real-world word relatedness, which can be important for many NLP applications. In this paper, we propose a model which extracts topics from text documents based on the common-sense knowledge available in ConceptNet [Speer et al., 2017] - a semantic concept graph that explicitly encodes real-world relations between words - and without any human supervision. When combining both ConceptNet’s knowledge graph and graph embeddings, our approach outperforms other baselines in the zero-shot setting, while generating a human-understandable explanation for its predictions through the knowledge graph. We study the importance of some modeling choices and criteria for designing the model, and we demonstrate that it can be used to label data for a supervised classifier to achieve an even better performance without relying on any humanly-annotated training data. We publish the code of our approach at https://github.com/D2KLab/ZeSTE and we provide a user friendly demo at https://zeste.tools.eurecom.fr/.

Subject Classification

ACM Subject Classification
  • Computing methodologies → Information extraction
Keywords
  • Topic Extraction
  • Zero-Shot Classification
  • Explainable NLP
  • Knowledge Graph

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Supplementary Materials

References

  1. Katherine Bailey and Sunny Chopra. Few-shot text classification with pre-trained word embeddings and a human in the loop. arXiv, 2018. URL: http://arxiv.org/abs/1804.02063.
  2. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems, pages 4349-4357, 2016. Google Scholar
  3. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal Sentence Encoder. arXiv, 2018. URL: http://arxiv.org/abs/1803.11175.
  4. Dawn Chen, Joshua C Peterson, and Thomas L Griffiths. Evaluating vector-space models of analogy. arXiv, 2017. URL: http://arxiv.org/abs/1705.04416.
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 4171-4186. Association for Computational Linguistics, 2019. Google Scholar
  6. Zakaria Elberrichi, Abdelattif Rahmoun, and Mohamed Amine Bentaalah. Using WordNet for Text Categorization. International Arab Journal of Information Technology (IAJIT), 5(1), 2008. Google Scholar
  7. Charles Elkan and Russell Greiner. Building large knowledge-based systems: Representation and inference in the Cyc project. Artificial Intelligence, 61(1):41-52, 1993. Google Scholar
  8. Ingo Feinerer and Kurt Hornik. wordnet: WordNet Interface, 2017. R package version 0.1-14. URL: https://CRAN.R-project.org/package=wordnet.
  9. Derek Greene and Pádraig Cunningham. Practical solutions to the problem of diagonal dominance in kernel document clustering. In 23^rd International Conference on Machine learning (ICML), pages 377-384, 2006. Google Scholar
  10. Antonio Gulli. AG’s corpus of news articles, 2005. URL: http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html.
  11. Yoon Kim. Convolutional neural networks for sentence classification. arXiv, 2014. URL: http://arxiv.org/abs/1408.5882.
  12. Ken Lang. Newsweeder: Learning to filter netnews. In 12^th International Conference on Machine Learning (ICML), pages 331-339, 1995. Google Scholar
  13. Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. On Measuring Social Biases in Sentence Encoders. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 622-628. Association for Computational Linguistics, 2019. Google Scholar
  14. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119, 2013. Google Scholar
  15. Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. Bias in Word Embeddings. In International Conference on Fairness, Accountability and Transparency (FAT), pages 446-–457. Association for Computing Machinery, 2020. Google Scholar
  16. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In International Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, 2014. Google Scholar
  17. Pushpankar Kumar Pushp and Muktabh Mayank Srivastava. Train once, test anywhere: Zero-shot learning for text classification. arXiv, 2017. URL: http://arxiv.org/abs/1712.05972.
  18. Charlotte Rudnik, Thibault Ehrhart, Olivier Ferret, Denis Teyssou, Raphaël Troncy, and Xavier Tannier. Searching News Articles Using an Event Knowledge Graph Leveraged by Wikidata. In 5^th Wiki Workshop, pages 1232-1239, 2019. Google Scholar
  19. Vishal S Shirsat, Rajkumar S Jagdale, and Sachin N Deshmukh. Sentence level sentiment identification and calculation from news articles using machine learning techniques. In Computing, Communication and Signal Processing, pages 371-376. Springer, 2019. Google Scholar
  20. Push Singh, Thomas Lin, Erik T Mueller, Grace Lim, Travell Perkins, and Wan Li Zhu. Open mind common sense: Knowledge acquisition from the general public. In OTM Confederated International Conferences On the Move to Meaningful Internet Systems, pages 1223-1237, 2002. Google Scholar
  21. Konstantinos Skianis, Fragkiskos Malliaros, and Michalis Vazirgiannis. Fusing document, collection and label graph-based representations with word embeddings for text classification. In 12^th Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs), New Orleans, Louisiana, USA, 2018. Google Scholar
  22. Yangqiu Song, Shyam Upadhyay, Haoruo Peng, Stephen Mayhew, and Dan Roth. Toward any-language zero-shot topic classification of textual documents. Artificial Intelligence, 274:133-150, 2019. Google Scholar
  23. R. Speer and Joshua Chin. An Ensemble Method to Produce High-Quality Word Embeddings. arXiv, 2016. URL: http://arxiv.org/abs/1604.01692.
  24. Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In 31^st AAAI Conference on Artificial Intelligence, 2017. Google Scholar
  25. Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to Fine-Tune BERT for Text Classification? arXiv, 2019. URL: http://arxiv.org/abs/1905.05583.
  26. Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. Learning to rank answers on large online qa collections. In 46^th Annual Meeting of the Association for Computational Linguistics (ACL), pages 719-727, 2008. Google Scholar
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv, 2017. URL: http://arxiv.org/abs/1706.03762.
  28. Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. arXiv, 2019. URL: http://arxiv.org/abs/1902.07153.
  29. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pages 5753-5763, 2019. Google Scholar
  30. Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach. arXiv, 2019. URL: http://arxiv.org/abs/1909.00161.
  31. Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. Integrating semantic knowledge to tackle zero-shot text classification. arXiv, 2019. URL: http://arxiv.org/abs/1903.12626.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact