Here Is Google DeepMind’s New Research To Build Massive LLMs With A Mixture Of Million Experts

A deep dive into the development of the Mixture-of-a-Million-Experts (MoME) architecture, which outperforms traditional LLMs in performance and computational efficiency like never before

Image generated with DALL-E 3

There is an LLM war happening around us.

It might not be immediately obvious, but all big tech companies are in a rush to develop better LLMs that outperform existing ones.

Increase the model size, the dataset size, and the amount of compute, and voila, you have a better model than before.

Given this Scaling law, researchers at…