Document Open Access Logo

Reverse Derivative Categories

Authors Robin Cockett, Geoffrey Cruttwell, Jonathan Gallagher, Jean-Simon Pacaud Lemay, Benjamin MacAdam, Gordon Plotkin, Dorette Pronk

PDF

File

Thumbnail PDF
PDF
LIPIcs.CSL.2020.18.pdf
  • Filesize: 0.54 MB
  • 16 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation → Semantics and reasoning
  • Theory of computation → Program semantics
Keywords
  • Reverse Derivatives
  • Cartesian Reverse Differential Categories
  • Categorical Semantics
  • Cartesian Differential Categories
  • Dagger Categories
  • Automatic Differentiation

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

The reverse derivative is a fundamental operation in machine learning and automatic differentiation [Martín Abadi et al., 2015; Griewank, 2012]. This paper gives a direct axiomatization of a category with a reverse derivative operation, in a similar style to that given by [Blute et al., 2009] for a forward derivative. Intriguingly, a category with a reverse derivative also has a forward derivative, but the converse is not true. In fact, we show explicitly what a forward derivative is missing: a reverse derivative is equivalent to a forward derivative with a dagger structure on its subcategory of linear maps. Furthermore, we show that these linear maps form an additively enriched category with dagger biproducts.

Cite As Get BibTex

Robin Cockett, Geoffrey Cruttwell, Jonathan Gallagher, Jean-Simon Pacaud Lemay, Benjamin MacAdam, Gordon Plotkin, and Dorette Pronk. Reverse Derivative Categories. In 28th EACSL Annual Conference on Computer Science Logic (CSL 2020). Leibniz International Proceedings in Informatics (LIPIcs), Volume 152, pp. 18:1-18:16, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2020) https://doi.org/10.4230/LIPIcs.CSL.2020.18

Author Details

Robin Cockett
  • University of Calgary, Department of Computer Science, Canada
Geoffrey Cruttwell
  • Mount Allison University, Department of , Mathematics and Computer Science, Canada
Jonathan Gallagher
  • Dalhousie University, Department of , Mathematics and Statistics, Canada
Jean-Simon Pacaud Lemay
  • University of Oxford, Department of Computer Science, UK
Benjamin MacAdam
  • University of Calgary, Department of Computer Science, Canada
Gordon Plotkin
  • Google Research
Dorette Pronk
  • Dalhousie University, Department of , Mathematics and Statistics, Canada

Funding

  • Cockett, Robin: Partially supported by NSERC (Canada).
  • Cruttwell, Geoffrey: Partially supported by NSERC (Canada).
  • Gallagher, Jonathan: Supported in part by NSERC and AARMS (Canada).
  • Lemay, Jean-Simon Pacaud: Supported by Kellogg College, the Clarendon Fund, and the Oxford Google-DeepMind Graduate Scholarship (UK).
  • MacAdam, Benjamin: Partially supported by NSERC (Canada).
  • Plotkin, Gordon: Supported by ESPRC (UK).
  • Pronk, Dorette: Partially supported by NSERC (Canada).

Acknowledgements

We thank Robert Seely for participating in the discussion on reverse differentiation with us.

References

  1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org. URL: https://www.tensorflow.org/.
  2. R. Blute, R. Cockett, and R. Seely. Cartesian Differential Categories. Theory and Applications of Categories, 22:622-672, 2009. Google Scholar
  3. R. Blute, R. Cockett, and R. Seely. Cartesian Differential Storage Categories. Theory and Applications of Categories, 30(18):620-686, 2015. Google Scholar
  4. R.F. Blute, J.R.B. Cockett, and R.A.G. Seely. Differential categories. Mathematical structures in computer science, 16(6):1049-1083, 2006. Google Scholar
  5. F. Borceaux. Handbook of categorical algebra II. Cambridge University Press, 2008. Google Scholar
  6. Bruce Christianson. A Leibniz notation for automatic differentiation. In Recent Advances in Algorithmic Differentiation, volume 87 of Lecture Notes in Computational Science and Engineering, pages 1-9. Springer, 2012. Google Scholar
  7. J.R.B. Cockett, G.S.H. Cruttwell, and J.D. Gallagher. Differential restriction categories. Theory and applications of categories, 25(21):537-613, 2011. Google Scholar
  8. J.R.B. Cockett and Stephen Lack. Restriction categories I: categories of partial maps. Theoretical Computer Science, 270(1):223-259, 2002. URL: https://doi.org/10.1016/S0304-3975(00)00382-0.
  9. R. Cockett and G. Cruttwell. Differential structure, tangent structure, and SDG. Applied Categorical Structures, 22:331-417, 2014. Google Scholar
  10. Robin Cockett, Geoffrey Cruttwell, Jonathan Gallagher, Jean-Simon Lemay, Benjamin MacAdam, Gordon Plotkin, and Dorette Pronk. Reverse derivative categories. Arxiv, 2019. Google Scholar
  11. Conal Elliott. The simple essence of automatic differentiation. Proceedings of the ACM on Programming Languages, 2(ICFP):70, 2018. Google Scholar
  12. Andreas Griewank. Who invented the reverse mode of differentiation. Documenta Mathematica, Extra Volume ISMP, pages 389-400, 2012. Google Scholar
  13. B. Jacobs. Categorical logic and type theory. Number 141 in Studies in logic and the foundations of mathematics. Elsevier, 1999. Google Scholar
  14. Anders Kock. The dual fibration in elementary terms. arXiv e-prints, page arXiv:1501.01947, January 2015. URL: http://arxiv.org/abs/1501.01947.
  15. J-S P. Lemay. A Tangent Category Alternative to the Faa Di Bruno Construction. Theory and Applications of Categories, 33(35):1072-1110, 2018. Google Scholar
  16. S. Linnainmaa. Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics, 16:146-160, 1976. Google Scholar
  17. G. Plotkin. A Simple Differential Programming Language. MFPS 2018 Keynote Address, June 2018. Google Scholar
  18. David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by backpropagating errors. Cognitive modeling, 5:3, 1988. URL: https://www.cs.toronto.edu/hinton/naturebp.pdf.
  19. Peter Selinger. Dagger Compact Closed Categories and Completely Positive Maps. Electron. Notes Theor. Comput. Sci., 170:139-163, March 2007. URL: https://doi.org/10.1016/j.entcs.2006.12.018.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact