arxiv preprint - Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum

In this episode, we discuss Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum by Hadi Pouransari, Chun-Liang Li, Jen-Hao Rick Chang, Pavan Kumar Anasosalu Vasu, Cem Koc, Vaishaal Shankar, Oncel Tuzel. The paper introduces a novel variable sequence length training technique called dataset decomposition to address inefficiencies in training large language models (LLMs) with fixed-length token sequences. It divides the dataset into buckets of sequences of the same size from unique documents and samples from these buckets with a curriculum during training, leading to computational savings and higher efficiency. This approach achieves target accuracy three times faster than traditional methods and enhances performance on standard language evaluations and long-context benchmarks.