RNNs Are Coming Back To Take Over Transformers (Yes, For Real!)

A deep dive into what RNNs, LSTMs, and GRUs are and how they are being modified to overcome the limitations of the currently prevalent Transformer architecture.

Image generated with DALL-E 3

Transformers have transformed our world.

They are the dominant AI architecture for almost all sequential tasks today.

One reason the Transformer architecture did so well is its Self-attention mechanism, which allows the processing of tokens simultaneously rather than sequentially as in previous architectures such as RNN, LSTM, and GRU.