Document Open Access Logo

Finally, a Polymorphic Linear Algebra Language (Pearl)

Authors Amir Shaikhha, Lionel Parreaux

PDF
Thumbnail PDF

File

LIPIcs.ECOOP.2019.25.pdf
  • Filesize: 498 kB
  • 29 pages

Document Identifiers

Author Details

Amir Shaikhha
  • Department of Computer Science, University of Oxford, UK
Lionel Parreaux
  • DATA Lab, EPFL, Lausanne, Switzerland

Acknowledgements

The authors would like to thank Jeremy Gibbons and Oleg Kiselyov for their helpful comments on draft versions of this paper. The first author was partially supported by EPFL during the preparation of this paper.

Cite AsGet BibTex

Amir Shaikhha and Lionel Parreaux. Finally, a Polymorphic Linear Algebra Language (Pearl). In 33rd European Conference on Object-Oriented Programming (ECOOP 2019). Leibniz International Proceedings in Informatics (LIPIcs), Volume 134, pp. 25:1-25:29, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2019)
https://doi.org/10.4230/LIPIcs.ECOOP.2019.25

Abstract

Many different data analytics tasks boil down to linear algebra primitives. In practice, for each different type of workload, data scientists use a particular specialised library. In this paper, we present Pilatus, a polymorphic iterative linear algebra language, applicable to various types of data analytics workloads. The design of this domain-specific language (DSL) is inspired by both mathematics and programming languages: its basic constructs are borrowed from abstract algebra, whereas the key technology behind its polymorphic design uses the tagless final approach (a.k.a. polymorphic embedding/object algebras). This design enables us to change the behaviour of arithmetic operations to express matrix algebra, graph algorithms, logical probabilistic programs, and differentiable programs. Crucially, the polymorphic design of Pilatus allows us to use multi-stage programming and rewrite-based optimisation to recover the performance of specialised code, supporting fixed sized matrices, algebraic optimisations, and fusion.

Subject Classification

ACM Subject Classification
  • Software and its engineering → Domain specific languages
  • Computing methodologies → Linear algebra algorithms
  • Mathematics of computing → Automatic differentiation
  • Mathematics of computing → Graph algorithms
  • Theory of computation → Probabilistic computation
Keywords
  • Linear Algebra
  • Domain-Specific Languages
  • Tagless Final
  • Polymorphic Embedding
  • Object Algebra
  • Multi-Stage Programming
  • Graph Processing
  • Probabilistic Programming
  • Automatic Differentiation

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. TensorFlow: A System for Large-Scale Machine Learning. In OSDI, volume 16, pages 265-283, 2016. Google Scholar
  2. Johan Anker and Josef Svenningsson. An EDSL approach to high performance Haskell programming. In ACM Haskell Symposium, pages 1-12, 2013. Google Scholar
  3. Emil Axelsson, Koen Claessen, Mary Sheeran, Josef Svenningsson, David Engdal, and Anders Persson. The Design and Implementation of Feldspar an Embedded Language for Digital Signal Processing. In Proceedings of the 22Nd International Conference on Implementation and Application of Functional Languages, IFL'10, pages 121-136, Berlin, Heidelberg, 2011. Springer-Verlag. Google Scholar
  4. Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. arXiv preprint, 2015. URL: http://arxiv.org/abs/1502.05767.
  5. Atilim Gunes Baydin, Barak A Pearlmutter, and Jeffrey Mark Siskind. DiffSharp: Automatic Differentiation Library. arXiv preprint, 2015. URL: http://arxiv.org/abs/1511.07727.
  6. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: A CPU and GPU math compiler in Python. In Proc. 9th Python in Science Conf, pages 1-7, 2010. Google Scholar
  7. Christian Bischof, Peyvand Khademi, Andrew Mauer, and Alan Carle. ADIFOR 2.0: Automatic differentiation of Fortran 77 programs. IEEE Computational Science and Engineering, 3(3):18-32, 1996. Google Scholar
  8. Christian H Bischof, HM Bucker, Bruno Lang, Arno Rasch, and Andre Vehreschild. Combining source transformation and operator overloading techniques to compute derivatives for MATLAB programs. In Source Code Analysis and Manipulation, 2002. Proceedings. Second IEEE International Workshop on, pages 65-72. IEEE, 2002. Google Scholar
  9. Jacques Carette and Oleg Kiselyov. Multi-stage Programming with Functors and Monads: Eliminating Abstraction Overhead from Generic Code. Sci. Comput. Program., 76(5):349-375, May 2011. Google Scholar
  10. Jacques Carette, Oleg Kiselyov, and Chung-Chieh Shan. Finally tagless, partially evaluated: Tagless staged interpreters for simpler typed languages. Journal of Functional Programming, 19(05):509-543, 2009. Google Scholar
  11. Bob Carpenter, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. Stan: A probabilistic programming language. Journal of statistical software, 76(1), 2017. Google Scholar
  12. Koen Claessen, Mary Sheeran, and Bo Joel Svensson. Expressive Array Constructs in an Embedded GPU Kernel Programming Language. In Proceedings of the 7th Workshop on Declarative Aspects and Applications of Multicore Programming, DAMP '12, pages 21-30, NY, USA, 2012. ACM. Google Scholar
  13. Duncan Coutts, Roman Leshchinskiy, and Don Stewart. Stream Fusion: From Lists to Streams to Nothing at All. In Proceedings of the 12th ACM SIGPLAN International Conference on Functional Programming, ICFP '07, pages 315-326, New York, NY, USA, 2007. ACM. Google Scholar
  14. Olivier Danvy and Andrzej Filinski. Abstracting control. In Proceedings of the 1990 ACM conference on LISP and functional programming, pages 151-160. ACM, 1990. Google Scholar
  15. Luc De Raedt, Angelika Kimmig, and Hannu Toivonen. ProbLog: A Probabilistic Prolog and Its Application in Link Discovery. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, IJCAI'07, pages 2468-2473, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc. Google Scholar
  16. Rina Dechter. Bucket elimination: A unifying framework for probabilistic inference. In Learning in graphical models, pages 75-104. Springer, 1998. Google Scholar
  17. Stephen Dolan. Fun with Semirings: A Functional Pearl on the Abuse of Linear Algebra. In Proceedings of the 18th ACM SIGPLAN International Conference on Functional Programming, ICFP '13, pages 101-110, New York, NY, USA, 2013. ACM. Google Scholar
  18. Conal M. Elliott. Beautiful Differentiation. In Proceedings of the 14th ACM SIGPLAN International Conference on Functional Programming, ICFP '09, pages 191-202, New York, NY, USA, 2009. ACM. Google Scholar
  19. Martin Erwig and Steve Kollmansberger. Functional pearls: Probabilistic functional programming in Haskell. Journal of Functional Programming, 16(1):21-34, 2006. Google Scholar
  20. Cormac Flanagan, Amr Sabry, Bruce F. Duba, and Matthias Felleisen. The Essence of Compiling with Continuations. In Proceedings of the ACM SIGPLAN 1993 Conference on Programming Language Design and Implementation, PLDI '93, pages 237-247, New York, NY, USA, 1993. ACM. Google Scholar
  21. Shaun A Forth. An efficient overloaded implementation of forward mode automatic differentiation in MATLAB. ACM Transactions on Mathematical Software (TOMS), 32(2):195-222, 2006. Google Scholar
  22. Franz Franchetti, Frédéric de Mesmay, Daniel McFarlin, and Markus Püschel. Operator language: A program generation framework for fast kernels. In Domain-Specific Languages, pages 385-409. Springer, 2009. Google Scholar
  23. Franz Franchetti, Yevgen Voronenko, and Markus Püschel. Formal Loop Merging for Signal Transforms. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '05, pages 315-326, 2005. Google Scholar
  24. Wally R Gilks, Andrew Thomas, and David J Spiegelhalter. A language and program for complex Bayesian modelling. The Statistician, pages 169-177, 1994. Google Scholar
  25. Andrew Gill, John Launchbury, and Simon L Peyton Jones. A short cut to deforestation. In Proceedings of the conference on Functional programming languages and computer architecture, FPCA, pages 223-232. ACM, 1993. Google Scholar
  26. Michele Giry. A categorical approach to probability theory. In Categorical aspects of topology and analysis, pages 68-85. Springer, 1982. Google Scholar
  27. Noah Goodman, Vikash Mansinghka, Daniel M Roy, Keith Bonawitz, and Joshua B Tenenbaum. Church: a language for generative models. arXiv preprint, 2012. URL: http://arxiv.org/abs/1206.3255.
  28. Andrew D Gordon, Thomas A Henzinger, Aditya V Nori, and Sriram K Rajamani. Probabilistic programming. In Proceedings of the on Future of Software Engineering, pages 167-181. ACM, 2014. Google Scholar
  29. Laurent Hascoet and Valérie Pascual. The Tapenade Automatic Differentiation Tool: Principles, Model, and Specification. ACM Trans. Math. Softw., 39(3):20:1-20:43, May 2013. Google Scholar
  30. Christian Hofer, Klaus Ostermann, Tillmann Rendel, and Adriaan Moors. Polymorphic embedding of DSLs. In Proceedings of the 7th international conference on Generative programming and component engineering, pages 137-148. ACM, 2008. Google Scholar
  31. Robin J. Hogan. Fast Reverse-Mode Automatic Differentiation Using Expression Templates in C++. ACM Trans. Math. Softw., 40(4):26:1-26:16, July 2014. Google Scholar
  32. Paul Hudak. Building Domain-specific Embedded Languages. ACM Comput. Surv., 28(4es), December 1996. Google Scholar
  33. Manohar Jonnalagedda and Sandro Stucki. Fold-based Fusion As a Library: A Generative Programming Pearl. In Proceedings of the 6th ACM SIGPLAN Symposium on Scala, pages 41-50. ACM, 2015. Google Scholar
  34. Jerzy Karczmarczuk. Functional differentiation of computer programs. ACM SIGPLAN Notices, 34(1):195-203, 1999. Google Scholar
  35. Gershon Kedem. Automatic Differentiation of Computer Programs. ACM Trans. Math. Softw., 6(2):150-165, June 1980. Google Scholar
  36. Oleg Kiselyov. Typed tagless final interpreters. In Generic and Indexed Programming, pages 130-174. Springer, 2012. Google Scholar
  37. Oleg Kiselyov. Reconciling Abstraction with High Performance: A MetaOCaml approach. Foundations and Trends in Programming Languages, 5(1):1-101, 2018. Google Scholar
  38. Oleg Kiselyov, Aggelos Biboudis, Nick Palladinos, and Yannis Smaragdakis. Stream Fusion, to Completeness. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages, POPL 2017, pages 285-299, New York, NY, USA, 2017. ACM. Google Scholar
  39. Oleg Kiselyov and Chung-Chieh Shan. Embedded probabilistic programming. In Domain-Specific Languages, pages 360-384. Springer, 2009. Google Scholar
  40. Tejas D Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. Picture: A probabilistic programming language for scene perception. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 4390-4399, 2015. Google Scholar
  41. Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: Effortless Gradients in Numpy. In ICML 2015 AutoML Workshop, 2015. Google Scholar
  42. Vikash K Mansinghka, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. Probabilistic programming with programmable inference. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 603-616. ACM, 2018. Google Scholar
  43. Brian Milch, Bhaskara Marthi, Stuart Russell, David Sontag, Daniel L Ong, and Andrey Kolobov. BLOG: Probabilistic Models with Unknown Objects. Statistical relational learning, page 373, 2007. Google Scholar
  44. Tom Minka, John Winn, John Guiver, and David Knowles. Infer.NET 2.4, 2010. Microsoft Research Cambridge, 2014. Google Scholar
  45. Mehryar Mohri. Semiring frameworks and algorithms for shortest-distance problems. Journal of Automata, Languages and Combinatorics, 7(3):321-350, 2002. Google Scholar
  46. Sri Hari Krishna Narayanan, Boyana Norris, and Beata Winnicka. ADIC2: Development of a component source transformation system for differentiating C and C++. Procedia Computer Science, 1(1):1845-1853, 2010. Google Scholar
  47. Georg Ofenbeck, Tiark Rompf, Alen Stojanov, Martin Odersky, and Markus Püschel. Spiral in Scala: Towards the Systematic Construction of Generators for Performance Libraries. In Proceedings of the 12th International Conference on Generative Programming: Concepts and Experiences, GPCE '13, pages 125-134, New York, NY, USA, 2013. ACM. Google Scholar
  48. Bruno C.d.S Oliveira and William R Cook. Extensibility for the Masses. In European Conference on Object-Oriented Programming, pages 2-27. Springer, 2012. Google Scholar
  49. Bruno C.d.S. Oliveira, Adriaan Moors, and Martin Odersky. Type Classes As Objects and Implicits. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA '10, pages 341-360, New York, NY, USA, 2010. ACM. Google Scholar
  50. Lionel Parreaux, Amir Shaikhha, and Christoph E. Koch. Quoted Staged Rewriting: A Practical Approach to Library-defined Optimizations. In Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences, GPCE 2017, pages 131-145, New York, NY, USA, 2017. ACM. Google Scholar
  51. Lionel Parreaux, Amir Shaikhha, and Christoph E. Koch. Squid: Type-safe, Hygienic, and Reusable Quasiquotes. In Proceedings of the 8th ACM SIGPLAN International Symposium on Scala, SCALA 2017, pages 56-66, New York, NY, USA, 2017. ACM. Google Scholar
  52. Lionel Parreaux, Antoine Voizard, Amir Shaikhha, and Christoph E. Koch. Unifying Analytic and Statically-typed Quasiquotes. Proc. ACM Program. Lang., 2(POPL):13:1-13:33, December 2017. Google Scholar
  53. Barak A Pearlmutter and Jeffrey Mark Siskind. Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. ACM Transactions on Programming Languages and Systems (TOPLAS), 30(2):7, 2008. Google Scholar
  54. Avi Pfeffer. Figaro: An object-oriented probabilistic programming language. Technical Report 137, Charles River Analytics, 2009. Google Scholar
  55. Markus Puschel, José MF Moura, Jeremy R Johnson, David Padua, Manuela M Veloso, Bryan W Singer, Jianxin Xiong, Franz Franchetti, Aca Gacic, Yevgen Voronenko, et al. SPIRAL: code generation for DSP transforms. Proceedings of the IEEE, 93(2):232-275, 2005. Google Scholar
  56. R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2014. URL: http://www.R-project.org/.
  57. Tiark Rompf and Martin Odersky. Lightweight Modular Staging: A Pragmatic Approach to Runtime Code Generation and Compiled DSLs. In the ninth international conference on Generative programming and component engineering, GPCE '10, pages 127-136, New York, NY, USA, 2010. ACM. Google Scholar
  58. Taisuke Sato. A Glimpse of Symbolic-Statistical Modeling by PRISM. Journal of Intelligent Information Systems, 31(2):161-176, October 2008. Google Scholar
  59. Amir Shaikhha, Mohammad Dashti, and Christoph Koch. Push versus Pull-Based Loop Fusion in Query Engines. Journal of Functional Programming, 28:e10, 2018. Google Scholar
  60. Amir Shaikhha, Andrew Fitzgibbon, Simon Peyton Jones, and Dimitrios Vytiniotis. Destination-passing Style for Efficient Memory Management. In Proceedings of the 6th ACM SIGPLAN International Workshop on Functional High-Performance Computing, FHPC 2017, pages 12-23, New York, NY, USA, 2017. ACM. Google Scholar
  61. Amir Shaikhha, Andrew Fitzgibbon, Dimitrios Vytiniotis, Simon Peyton Jones, and Christoph Koch. Efficient Differentiable Programming in a Functional Array-Processing Language. arXiv preprint, 2018. URL: http://arxiv.org/abs/1806.02136.
  62. Daniele G. Spampinato and Markus Püschel. A Basic Linear Algebra Compiler. In Proceedings of Annual IEEE/ACM International Symposium on Code Generation and Optimization, CGO '14, pages 23:23-23:32. ACM, 2014. Google Scholar
  63. Josef Svenningsson. Shortcut Fusion for Accumulating Parameters & Zip-like Functions. In Proceedings of the Seventh ACM SIGPLAN International Conference on Functional Programming, ICFP '02, pages 124-132. ACM, 2002. Google Scholar
  64. Bo Joel Svensson and Josef Svenningsson. Defunctionalizing Push Arrays. In Proceedings of the 3rd ACM SIGPLAN Workshop on Functional High-performance Computing, FHPC '14, pages 43-52, NY, USA, 2014. ACM. Google Scholar
  65. Walid Taha. A gentle introduction to multi-stage programming. In Domain-Specific Program Generation, pages 30-50. Springer, 2004. Google Scholar
  66. Walid Taha and Tim Sheard. MetaML and multi-stage programming with explicit annotations. Theor. Comput. Sci., 248(1-2):211-242, 2000. Google Scholar
  67. Philip Wadler. Deforestation: Transforming programs to eliminate trees. In ESOP'88, pages 344-358. Springer, 1988. Google Scholar
  68. Philip Wadler. The expression problem. Java-genericity mailing list, 1998. Google Scholar
  69. Fei Wang, James Decker, Xilun Wu, Gregory Essertel, and Tiark Rompf. Backpropagation with Callbacks: Foundations for Efficient and Expressive Differentiable Programming. In Advances in Neural Information Processing Systems, pages 10200-10211, 2018. Google Scholar
  70. Matthew J Weinstein and Anil V Rao. Algorithm 984: ADiGator, a toolbox for the algorithmic differentiation of mathematical functions in MATLAB using source transformation via operator overloading. ACM Trans. Math. Softw, 2016. Google Scholar
  71. Jianxin Xiong, Jeremy Johnson, Robert Johnson, and David Padua. SPL: A Language and Compiler for DSP Algorithms. In Proceedings of the ACM SIGPLAN 2001 Conference on Programming Language Design and Implementation, PLDI '01, pages 298-308, New York, NY, USA, 2001. ACM. Google Scholar
  72. Jeremy Yallop. Staged Generic Programming. Proc. ACM Program. Lang., 1(ICFP):29:1-29:29, August 2017. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact