ODSL Forum: "Enzyme: Automatic Differentiation of Scientific Simulations for Optimists"


Presenter: Ludger Paehler, TUM
Title: Enzyme: Automatic Differentiation of Scientific Simulations for Optimists​​​​​​​

Abstract: Computing derivatives is key to many algorithms in scientific computing and machine learning such as optimization, uncertainty quantification, and stability analysis. Reverse-mode AD can efficiently compute gradients by synthesizing a ”reverse-pass”, or also called adjoint, to the original function that requires only a single execution to compute the derivative of all input parameters. As a consequence reverse-mode AD has been the tool of choice for larger applications with many parameters. Enzyme is an LLVM compiler-plugin that performs reverse-mode automatic differentiation by synthesizing gradients of statically analyzable programs expressible in the LLVM intermediate representation such as C/C++, FORTRAN, Julia, Go, Swift, and Rust. Unlike traditional source-to- source and operator-overloading tools, Enyzme performs AD on an optimized IR, which has been shown to vastly outperform traditional AD approaches [1]. Through a combination of novel techniques we have been able to make Enzyme the first fully automatic reverse-mode AD tool to generate gradients of GPU kernels [2]. Recently we have further extended on Enzyme’s additive capabilities to also automatically differentiate programs utilizing MPI, OpenMP, and RAJA [3]. To showcase Enzyme’s capabilities, I will present detailed evaluations of Enzyme’s individual capabilities, as well as go into depth on Enzyme’s current limitations and the developments we are undertaking to extend Enzyme to ever bigger, and more complex programs.

[1] Moses, W., and Churavy, V. Instead of rewriting foreign code for machine learning, automatically synthesize fast gradients. Advances in Neural Information Processing Systems 33 (2020).

[2] Moses, W. S., Churavy, V., Paehler, L., Hückelheim, J., Narayanan, S. H. K., Schanen, M., and Doerfert, J. Reverse-mode automatic differentiation and optimization of gpu kernels via enzyme. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (New York, NY, USA, 2021), SC ’21, Association for Computing Machinery.

[3] Moses, W. S., Paehler, L., Churavy, V., Schanen, M., Narayanan, S. H. K., Hückelheim, J., and Doerfert, Johannes amd Hovland, P. Language-independent automatic differentiation across multiple parallel paradigms. (under review) (2022).

As usual, the format will be a short presentation followed by plenty of discussion.

Please let all interested know about the Journal Club (get them to send their Email address).  We are looking forward to lively discussions.