Step 1 — Ingestion
NewsIQ continuously fetches new articles from our curated publisher list via RSS feeds and news APIs. Articles are normalized, deduplicated by URL, and queued for analysis. Ingestion runs every 5 minutes.
Step 2 — Semantic Embedding
Each article's headline and lead paragraph is transformed into a high-dimensional semantic vector using a sentence transformer model. These embeddings capture the meaning of the text, not just keywords, allowing us to match articles about the same event even when they use different terminology.
Step 3 — Story Clustering
Articles within a rolling 72-hour window are clustered using cosine similarity on their embeddings, combined with named entity overlap (same people, organizations, and locations = higher relatedness score). Clusters that exceed our similarity threshold are merged into a single story with a shared headline.
Accuracy: 98% clustering precision on our internal benchmark of 10,000 story events. Evaluated against human editorial judgement.
Step 4 — AI Summarisation
Once a cluster is formed, our LLM pipeline generates:
- • A neutral headline rewritten from the cluster consensus
- • A one-line summary (the single most important fact)
- • A short summary (2–3 sentences of context)
- • A detailed summary with key facts, entities, and implications
- • Extracted key facts as a structured list
All outputs are grounded in the retrieved article content. The LLM cannot introduce information not present in the source articles.
Step 5 — Difference Engine
For each source within a cluster, the Difference Engine identifies: what unique information this source covers, what information it omits compared to the cluster consensus, and where it factually contradicts other sources. This gives readers genuine cross-source literacy.
Step 6 — Timeline Construction
For developing stories, our system extracts timestamped events from article content and orders them chronologically. Timelines are rebuilt each time new articles are added to the cluster.
Frequently Asked Questions
How does NewsIQ cluster related news articles?
NewsIQ uses a combination of semantic embedding similarity, named entity overlap, and topic modeling to group articles about the same real-world event into a single story cluster. Articles published within a rolling time window are scored for relatedness and merged when the similarity score exceeds our threshold.
How accurate is the NewsIQ AI clustering?
Our internal evaluation shows 98% clustering accuracy on held-out test sets of major news events. False positives (wrongly merging unrelated stories) are rare due to our conservative similarity threshold. False negatives (missing related articles) are addressed by the continuous ingestion cycle.
How often does NewsIQ update its story feed?
NewsIQ ingests new articles every 5 minutes. Trending stories are updated with higher priority. Story summaries are regenerated when a significant new source is added to the cluster.
Does NewsIQ use GPT or other LLMs?
NewsIQ uses large language models for headline generation, summary creation, entity extraction, and timeline construction. The specific models used are subject to change as we improve quality. All LLM outputs are grounded in retrieved source article content — we do not generate free-form content.