Semantic Deduplication: Finding Near-Duplicate Content by Meaning

Semantic deduplication is a technique for detecting near-duplicate content by comparing *meaning* rather than exact text. Documents are converted to dense vector embeddings using transformer models; pairs with cosine similarity above a threshold (typically 0.85–0.95) are flagged as potential duplicates. ## Why It's Better Than Hash-Based Dedup Traditional deduplication (content hashing, exact string matching) misses: - **Paraphrased content**: Same information expressed differently - **Translated content**: Same meaning in different languages - **Reformatted content**: Same text with different structure or whitespace - **Near-duplicates**: 90% identical with minor edits Semantic dedup catches all of these because it operates on meaning, not surface text. ## Implementation The typical pipeline: 1. Generate embeddings for all documents using a model (e.g., sentence-transformers, OpenAI embeddings) 2. Store in a pgvector: PostgreSQL Extension for Vector Similarity Search 3. For each new document, query for nearest neighbors above the similarity threshold 4. Flag or merge flagged duplicates ## Threshold Selection Higher thresholds (0.95+) catch only very close duplicates; lower thresholds (0.80–0.85) are more aggressive but risk false positives on merely related (not duplicate) content. The right threshold depends on content type and tolerance for missed duplicates vs. false merges. **See also:** Vector Databases: How Embedding Search Powers Modern AI Applications