Step 1 — Ingestion
NewsIQ continuously fetches new articles from our curated publisher list via RSS feeds and news APIs. Articles are normalized, deduplicated by URL, and queued for analysis. Ingestion runs every 15 minutes.
Step 2 — Semantic Embedding
Each article's headline and lead paragraph is transformed into a high-dimensional semantic vector using a sentence transformer model. These embeddings capture the meaning of the text, not just keywords, allowing us to match articles about the same event even when they use different terminology.
Step 3 — Story Clustering
Articles within a rolling 72-hour window are clustered using cosine similarity on their embeddings, combined with named entity overlap (same people, organizations, and locations = higher relatedness score). Clusters that exceed our similarity threshold are merged into a single story with a shared headline.
Accuracy: thresholds are conservative, so NewsIQ would rather show two stories about one event than merge two different events. A scheduled job merges duplicate stories afterwards.
Step 4 — AI Summarisation
Once a cluster is formed, our LLM pipeline generates:
- • A neutral headline rewritten from the cluster consensus
- • A one-line summary (the single most important fact)
- • A short summary (2–3 sentences of context)
- • A detailed summary with key facts, entities, and implications
- • Extracted key facts as a structured list
All outputs are grounded in the retrieved article content. The LLM cannot introduce information not present in the source articles.
Step 5 — Difference Engine
For each source within a cluster, the Difference Engine identifies: what unique information this source covers, what information it omits compared to the cluster consensus, and where it factually contradicts other sources. This gives readers genuine cross-source literacy.
Step 6 — Timeline Construction
For developing stories, our system extracts timestamped events from article content and orders them chronologically. Timelines are rebuilt each time new articles are added to the cluster.
Frequently Asked Questions
How does NewsIQ cluster related news articles?
NewsIQ uses a combination of semantic embedding similarity and named-entity overlap to group articles about the same real-world event into a single story cluster. Articles published within a rolling time window are scored for relatedness and merged when the similarity score exceeds our threshold.
How accurate is the NewsIQ AI clustering?
Clustering uses deliberately conservative similarity thresholds, so related articles are sometimes left in separate stories rather than unrelated ones being merged. A scheduled job later merges stories that turn out to describe the same event.
How often does NewsIQ update its story feed?
NewsIQ checks publisher feeds for new articles every 15 minutes. Story summaries are regenerated when a significant new source is added to the cluster.
Does NewsIQ use GPT or other LLMs?
NewsIQ uses large language models for headline generation, summary creation, entity extraction, and timeline construction. The specific models used are subject to change as we improve quality. All LLM outputs are grounded in retrieved source article content — we do not generate free-form content.