Document Open Access Logo

Generalizing Deep Models for Overhead Image Segmentation Through Getis-Ord Gi* Pooling

Authors Xueqing Deng, Yuxin Tian, Shawn Newsam

PDF
Thumbnail PDF

File

LIPIcs.GIScience.2021.I.3.pdf
  • Filesize: 1.13 MB
  • 14 pages

Document Identifiers

Author Details

Xueqing Deng
  • EECS, University of California, Merced, CA, USA
Yuxin Tian
  • EECS, University of California, Merced, CA, USA
Shawn Newsam
  • EECS, University of California, Merced, CA, USA

Funding

This work was funded in part by a National Science Foundation grant, #IIS-1747535.

Acknowledgements

We gratefully acknowledge the support of NVIDIA Corporation through the donation of the GPU card used in this work. And we thank Yi Zhu for providing helpful discussion.

Cite AsGet BibTex

Xueqing Deng, Yuxin Tian, and Shawn Newsam. Generalizing Deep Models for Overhead Image Segmentation Through Getis-Ord Gi* Pooling. In 11th International Conference on Geographic Information Science (GIScience 2021) - Part I. Leibniz International Proceedings in Informatics (LIPIcs), Volume 177, pp. 3:1-3:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2020)
https://doi.org/10.4230/LIPIcs.GIScience.2021.I.3

Abstract

That most deep learning models are purely data driven is both a strength and a weakness. Given sufficient training data, the optimal model for a particular problem can be learned. However, this is usually not the case and so instead the model is either learned from scratch from a limited amount of training data or pre-trained on a different problem and then fine-tuned. Both of these situations are potentially suboptimal and limit the generalizability of the model. Inspired by this, we investigate methods to inform or guide deep learning models for geospatial image analysis to increase their performance when a limited amount of training data is available or when they are applied to scenarios other than which they were trained on. In particular, we exploit the fact that there are certain fundamental rules as to how things are distributed on the surface of the Earth and these rules do not vary substantially between locations. Based on this, we develop a novel feature pooling method for convolutional neural networks using Getis-Ord Gi* analysis from geostatistics. Experimental results show our proposed pooling function has significantly better generalization performance compared to a standard data-driven approach when applied to overhead image segmentation.

Subject Classification

ACM Subject Classification
  • Computing methodologies → Neural networks
Keywords
  • Remote sensing
  • convolutional neural networks
  • pooling function
  • semantic segmentation
  • generalization

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. ISPRS 2D Semantic Labeling Challenge. URL: http://www2.isprs.org/commissions/comm3/wg4/semantic-labeling.html.
  2. N. Audebert, B. Saux, and S. Lefèvre. Beyond RGB: Very High Resolution Urban Remote Sensing with Multimodal Deep Networks. ISPRS Journal of Photogrammetry and Remote Sensing, 2018. Google Scholar
  3. V. Badrinarayanan, A. Kendall, and R. Cipolla. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017. Google Scholar
  4. J. Chen, C. Wang, A. Yue, J. Chen, D. He, and X. Zhang. Knowledge-Guided Golf Course Detection Using a Convolutional Neural Network Fine-Tuned on Temporally Augmented Data. J. Appl. Remote Sens., 2017. Google Scholar
  5. L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. Yuille. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017. Google Scholar
  6. L. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv preprint arXiv:1706.05587, 2017. Google Scholar
  7. L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. In European conference on computer vision (ECCV), 2018. Google Scholar
  8. X. Chen, S. Xiang, C. Liu, and C. Pan. Vehicle Detection in Satellite Images by Hybrid Deep Convolutional Neural Networks. IEEE Geoscience and Remote Sensing Letters, 2014. Google Scholar
  9. Y. Chen, W. Chen, Y. Chen, B. Tsai, Y. Wang, and M. Sun. No More Discrimination: Cross City Adaptation of Road Scene Segmenters. In International Conference on Computer Vision (ICCV), 2017. Google Scholar
  10. X. Deng, H. L. Yang, N. Makkar, and D. Lunga. Large Scale Unsupervised Domain Adaptation of Segmentation Networks with Adversarial Learning. In IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2019. Google Scholar
  11. J. Hoffman, E. Tzeng, T. Park, J. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell. CyCADA: Cycle-Consistent Adversarial Domain Adaptation. In International Conference on Machine Learning (ICML), 2018. Google Scholar
  12. J. Hoffman, D. Wang, F. Yu, and T. Darrell. FCNs in the Wild: Pixel-Level Adversarial and Constraint-based Adaptation. arXiv preprint arXiv:1612.02649, 2016. Google Scholar
  13. G. Huang, Z. Liu, L. Van Der Maaten, and K. Weinberger. Densely Connected Convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. Google Scholar
  14. A. Karpatne, W. Watkins, J. Read, and V. Kumar. Physics-Guided Neural Networks (PGNN): An Application in Lake Temperature Modeling. arXiv preprint arXiv:1710.11431, 2017. Google Scholar
  15. J. Kuen, X. Kong, G. Wang, and Y. Tan. DelugeNets: Deep Networks with Efficient and Flexible Cross-Layer Information Inflows. In International Conference on Computer Vision (ICCV), 2017. Google Scholar
  16. N. Kussul, M. Lavreniuk, S. Skakun, and A. Shelestov. Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data. IEEE Geoscience and Remote Sensing Letters, 2017. Google Scholar
  17. C. Lee, P. Gallagher, and Z. Tu. Generalizing Pooling Functions in Convolutional Neural Nnetworks: Mixed, Gated, and Tree. In Artificial Intelligence and Statistics, 2016. Google Scholar
  18. Y. Liu, B. Fan, L. Wang, J. Bai, S. Xiang, and C. Pan. Semantic Labeling in Very High Resolution Images via a Self-Cascaded Convolutional Neural Network. ISPRS Journal of Photogrammetry and Remote Sensing, 2018. Google Scholar
  19. J. Long, E. Shelhamer, and T. Darrell. Fully Convolutional Networks for Semantic Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. Google Scholar
  20. X. Ma, Z. Dai, Z. He, J. Ma, Y. Wang, and Y. Wang. Learning Traffic as Images: A Deep Convolutional Neural Network for Large-Scale Transportation Network Speed Prediction. Sensors, 2017. Google Scholar
  21. E. Maggiori, Y. Tarabalka, G. Charpiat, and P. Alliez. High-Resolution Aerial Image Labeling with Convolutional Neural Networks. IEEE Transactions on Geoscience and Remote Sensing, 2017. Google Scholar
  22. V. Mnih and G. E. Hinton. Learning to Detect Roads in High-Resolution Aerial Images. In European Conference on Computer Vision (ECCV), 2010. Google Scholar
  23. L. Mou, Y. Hua, and X. X. Zhu. A Relation-Augmented Fully Convolutional Network for Semantic Segmentation in Aerial Scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Google Scholar
  24. J. K. Ord and Arthur Getis. Local Spatial Autocorrelation Statistics: Distributional Issues and an Application. Geographical Analysis, 1995. Google Scholar
  25. O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional Networks for Biomedical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015. Google Scholar
  26. F. Saeedan, N. Weber, M. Goesele, and S. Roth. Detail-preserving pooling in deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. Google Scholar
  27. Z. Shao, W. Zhou, X. Deng, M. Zhang, and Q. Cheng. Multilabel Remote Sensing Image Retrieval Based on Fully Convolutional Network. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020. Google Scholar
  28. Jamie Sherrah. Fully Convolutional Networks for Dense Semantic Labelling of High-Resolution Aerial Imagery. arXiv preprint arXiv:1606.02585, 2016. Google Scholar
  29. K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556, 2014. Google Scholar
  30. J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for Simplicity: The All Convolutional Net. In International Conference on Learning Representation workshop (ICLR workshop), 2015. Google Scholar
  31. Y. Tian, X. Deng, Y. Zhu, and S. Newsam. Cross-Time and Orientation-Invariant Overhead Image Geolocalization Using Deep Local Features. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2020. Google Scholar
  32. W. R. Tobler. A Computer Movie Simulating Urban Growth in the Detroit Region. Economic Geography, 1970. Google Scholar
  33. Y. Tsai, W. Hung, S. Schulter, K. Sohn, M. Yang, and M. Chandraker. Learning to Adapt Structured Output Space for Semantic Segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. Google Scholar
  34. Y. Tsai, K. Sohn, S. Schulter, and M. Chandraker. Domain Adaptation for Structured Output via Discriminative Representations. In International conference on Computer Vision (ICCV), 2019. Google Scholar
  35. E. Tzeng, J. Hoffman, K. Saenko, and T. Darrell. Adversarial Discriminative Domain Adaptation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. Google Scholar
  36. Z. Wei, J. Zhang, L. Liu, F. Zhu, F. Shen, Y. Zhou, S. Liu, Y. Sun, and L. Shao. Building Detail-Sensitive Semantic Segmentation Networks with Polynomial Pooling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Google Scholar
  37. J. Yuan. Learning Building Extraction in Aerial Scenes with Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2017. Google Scholar
  38. S. Zagoruyko and N. Komodakis. Wide Residual Networks. arXiv preprint arXiv:1605.07146, 2016. Google Scholar
  39. P. Zhang, X. Niu, Y. Dou, and F. Xia. Airport Detection from Remote Sensing Images using Transferable Convolutional Neural Networks. In International Joint Conference on Neural Networks (IJCNN), 2016. Google Scholar
  40. H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid Scene Parsing Network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. Google Scholar
  41. X. X. Zhu, D. Tuia, L. Mou, G. Xia, L. Zhang, F. Xu, and F. Fraundorfer. Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources. IEEE Geoscience and Remote Sensing Magazine, 2017. Google Scholar
  42. Y. Zhu, K. Sapra, F. A. Reda, K. J. Shih, S. Newsam, A. Tao, and B. Catanzaro. Improving Semantic Segmentation via Video Propagation and Label Relaxation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact