The paper “CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training” presents an automated framework that iteratively improves the data mixture used for pre-training large language models by embedding and clustering large-scale unlabeled datasets into semantic groups, then evaluating model performance on these clusters to dynamically adjust sampling weights toward more informative or challenging data. The approach uses a smaller proxy model and a predictor to efficiently search the vast space of data mixtures without relying on explicit domain labels. Experiments training a 1-billion-parameter model on a 400-billion-token optimized mixture show a 2% performance gain over the state-of-the-art Llama-3.2-1B, with domain-specific optimization (e.g., Social Sciences) achieving up to 5% improvement. The paper includes detailed experimentation on reasoning benchmarks and introduces two new datasets—ClimbLab, a 1.2-trillion-token corpus clustered into 20 semantic groups, and ClimbMix, a compact 400-billion-token dataset optimized for efficient pre-training—demonstrating that CLIMB’s iterative, clustering-based refinement leads to superior model generalization and specialization under fixed compute budgets.