All the Transformer Math You Need to Know
Why I kept it
A reference on how transformers are scaled.
François Paupier
Google DeepMind4 February 2025Public source
Read the article
In this reference
Part 4 of How to Scale Your Model: how to calculate FLOPs, bytes, and other quantities for a Transformer.
From the chapter
Here we'll do a quick review of the Transformer architecture, specifically how to calculate FLOPs, bytes, and other quantities of interest.
Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, Charlie Chen, Sharad Vikram, Federico Lebron, Peter Choy, Vinay Ramasesh, Albert Webson, and Reiner Pope
Cite the original work
Jacob Austin & Sholto Douglas & Roy Frostig & Anselm Levskaya & Charlie Chen & Sharad Vikram & Federico Lebron & Peter Choy & Vinay Ramasesh & Albert Webson & Reiner Pope (4 February 2025). All the Transformer Math You Need to Know. Google DeepMind. https://jax-ml.github.io/scaling-book/transformers/
View BibTeX
@misc{all_the_transformer_math_you_need_to_know,
author = {Jacob Austin and Sholto Douglas and Roy Frostig and Anselm Levskaya and Charlie Chen and Sharad Vikram and Federico Lebron and Peter Choy and Vinay Ramasesh and Albert Webson and Reiner Pope},
title = {{All the Transformer Math You Need to Know}},
howpublished = {Google DeepMind},
url = {https://jax-ml.github.io/scaling-book/transformers/},
year = {2025}
}
