A Simple Principle From Noise-Cancelling Headphones Supercharges Transformers Like Never Before

A deep dive into the ‘Differential Transformer’ architecture, learning how it works and why it is such a promising architecture to advance LLMs.

Image generated with Google ImageFX

Nearly all popular LLMs today are based on the Decoder-only Transformer architecture.

At the heart of this architecture is the Transformer with its Attention mechanism.

This mechanism weighs the importance of different elements in an input sequence and adjusts their influence on the output using Attention scores.

But Transformers aren’t perfect.

They tend to over-allocate attention to irrelevant context.

Fortunately, we have a new technique that can be applied to the Transformer to fix this issue.

The resulting architecture, called the Differential Transformer, beats the conventional Transformer when the model size and training tokens are scaled.