News Scraping at Scale: Deduplicating Syndicated Stories and Extracting Clean Articles
This post builds the pipeline that turns raw news pages into a clean, deduplicated dataset: URL canonicalisation, article extraction, reliable dates, and near-duplicate clustering with MinHash. Every snippet below was run against sample pages. Where the duplicates come from Counting…