C4 (Colossal Clean Crawled Corpus)
C4 is a roughly 750-gigabyte English text dataset released by Google in 2019 alongside the T5 model, derived from a single April 2019 snapshot of Common Crawl after heuristic deduplication and quality filtering. It has been used to train T5, LaMDA, and many open-source LLMs, and is the subject of a widely-cited 2021 audit by Dodge et al.
C4 (Colossal Clean Crawled Corpus) is a large English-language text dataset created by Google researchers Colin Raffel and colleagues and released in 2019 alongside the *T5* (Text-to-Text Transfer Transformer) paper. It was constructed by taking a single April 2019 snapshot of Common Crawl and applying a sequence of heuristic filters: removing pages that did not end lines with terminal punctuation, removing pages shorter than three sentences, removing lines containing words from an obscenity blocklist, removing pages mentioning JavaScript, and deduplication via three-sentence overlap. After filtering, C4 contains roughly 156 billion tokens (about 750 GB of text), drawn from more than 365 million distinct domains. The largest contributing domains include patents.google.com, en.wikipedia.org, and a long tail of news and reference sites; Wikipedia itself contributes about 290 million tokens, or 0.2% of the dataset by raw token count, though it is typically upsampled at training time. C4 became a de facto standard corpus for English-only LLM pretraining: Google used C4 (and its multilingual variant mC4) to train T5, LaMDA (where it made up roughly 12.5% of training tokens), and later models; Meta and many open-source projects use it directly or as a starting point. Its public TensorFlow Datasets release made it the most accessible large pretraining corpus until newer datasets like RefinedWeb and The Pile appeared. A widely-cited 2021 audit by Jesse Dodge and colleagues — *Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus* — documented several problematic properties. The English-detection filter disproportionately removed text from African-American English and Hispanic-aligned dialects. The blocklist suppressed substantial LGBTQ+ content alongside actual obscenity. Machine-generated patent text was overrepresented. And copyrighted material from major news publishers appeared without licensing. These findings sparked ongoing debate about quality filtering as a representation harm vector and motivated more careful documentation of LLM training corpora — including the datasheet and data statement practices that have since become standard.