Reinforcement Learning On Pre-Training Data Improves LLMs Like Never Before

A deep dive into RLPT, a technique to RL train LLMs on the pre-training dataset without any need for human annotation for rewards.

Image generated with Google ImageFX and edited using Nano Banana