Fourier Analysis Networks (FANs) Are Here To Break Barriers In AI

A deep dive into Fourier Analysis Network (FAN), a novel neural network architecture that performs better than the baselines (MLP, KAN, Transformer and Mamba), and learning to build one from scratch.

Image generated with DALL-E 3

Multi-layer Perceptrons, or MLPs, are the dominant architecture for AI models today.

They are based on the Universal Approximation Theorem and aim to approximate any real continuous function to any desired accuracy with their hidden layers.

MLPs have been recently challenged by Kolmogorov-Arnold Networks (KANs) and XNets.

But there’s something that is still lacking in the core of these architectures.

They cannot model periodicity from data.

Therefore, their performance on periodic data remains poor.

This has been solved by a new type of neural network architecture called Fourier Analysis Networks (FANs).

Published in ArXiv, this research introduces FANs that use the principles of Fourier Analysis to encode periodic patterns directly within the neural network.

Surprisingly, FANs are exceptional at modelling periodic data compared to MLPs, KANs, and Transformers.

FANs outperform these baselines in symbolic formula representation (both periodic and non-periodic).

And, a FAN-based Transformer beats the MLP-based Transformer, LSTM, and Mamba for time series forecasting and language modeling tasks.

They do this all by still using fewer parameters and floating-point operations (FLOPs) than their counterparts.

Performance of different neural network architectures for the sine function (Image from the original research paper)

Here is a story where we discuss how FANs work, compare their performance with other popular neural network architectures, and then learn to build and train one from scratch.

Let’s begin!

---

Let’s Start With Fourier Analysis

Fourier Analysis is a mathematical way of breaking down functions into the frequencies that make them up.

The analysis is based on the Fourier Series, which represents a periodic function as an infinite sum of sine and cosine terms.

The Fourier series expansion for a periodic function f(x) where T is the function’s period is shown below — 

Fourier Series expansion of function f(x)

The coefficients a(n) and b(n) are calculated by integrating the function over one period as follows — 

Calculations for coefficients a(n) and b(n)

One of the best parts of the Fourier series is that it can even be extended to handle non-periodic functions.