Semantic Deduplication System for Real-Time News Aggregation.

Designed a real-time content pipeline using vector embeddings and similarity search to detect duplicate news articles at the semantic level, where URL-based filtering could not reliably identify overlapping content.

Overview

This project involved building a real-time news aggregation system for a UAE-based account, focused on sourcing and processing articles from multiple global and regional news sources.

A key requirement was ensuring duplicate content was not stored even when the same or near-identical article appeared across publications with different URLs.

The Challenge

• Identical or near-identical articles published across multiple sources
• Different URLs made traditional deduplication ineffective
• High volumes of incoming articles required real-time processing
• The database needed to remain clean and non-redundant

The core challenge was moving from URL-based filtering to content-level semantic deduplication.

Approach

Real-time ingestion pipeline

Designed a system to continuously ingest news articles from multiple sources.

Embedding generation

Converted article content into vector embeddings using OpenAI embedding models to create semantic representations of each article.

Vector database integration

Stored embeddings in Pinecone for efficient similarity search across a growing article dataset.

Semantic similarity detection

Implemented cosine-similarity-based search to compare new articles with existing content and identify duplicates based on meaning rather than URL identity.

Threshold-based filtering

Applied a configurable similarity threshold to determine whether incoming content should be stored or discarded.

What We Implemented

• Real-time news ingestion pipeline
• OpenAI-based embedding generation
• Pinecone vector database integration
• Semantic similarity search
• Threshold-based deduplication logic

Outcome

• Eliminated duplicate content at a semantic level
• Improved data quality and consistency
• Enabled real-time processing without manual intervention
• Built a scalable foundation for content ingestion and analysis

The result was a semantic deduplication system designed to ensure only unique content entered the database, even across overlapping publishers.

Engagement

We worked with a Bangalore-based agency for a UAE-based account to design and implement the system. The solution was designed to handle increasing data volumes while maintaining accurate content filtering and deduplication.