A Detailed History of Optimizers (And How The New ‘Adam-mini’ Optimizer Works)

A deep dive into how Optimizers work, the history of their development, and how the novel 'Adam-mini' optimizer enhances LLM training like never before

Image generated with DALL-E 3

An Optimizer forms the basis for training most modern neural networks.

Published in 2017, the Adam Optimizer, along with its variants, has become the dominant and go-to optimizer for training LLMs in the industry today.

But there’s an issue with Adam that has been largely overlooked due to its superior performance.

That issue i…