Skip to main content
Skip to main content
Helix
Book a briefing →

News Feeds Concepts

Core concepts and terminology for understanding the News Feeds API.

What is a News Feed?​

A news feed is a container that aggregates articles from multiple sources into a single stream. You create feeds, add sources and filters to them, and retrieve the collected articles through the API.

Core Entities​

Feed​

A feed is your primary resource. It holds configuration and serves as the container for all collected content.

Key properties:

  • Unique identifier
  • Name and description
  • Enabled/disabled state (a new feed starts disabled)
  • config: how long an item's images stay available (media.retentionDaysAfterAdded, 30 days by default)

A feed doesn't do anything on its own. You add sources to it to start collecting content, and filters to decide which articles it carries.

Source​

A source defines where content comes from. Each source on a feed tells the system what to monitor for it. You add websites through the API ({ "type": "site", "url": … }, the whole domain); our team adds the other types on request. You can list, enable, disable and remove every source on your feeds. See Sources.

Source types:

  • Site: Monitor an entire domain for articles
  • Sitemap: Track a specific XML sitemap for new URLs
  • Index Page: Watch a page that lists articles (e.g., a news homepage)
  • Instagram: Track posts and reels from an Instagram account
  • TikTok: Track videos from a TikTok account
  • Twitter / X: Track tweets from a Twitter / X account
  • Facebook: Track content from a Facebook page, group, or profile

When you add a source, the system begins monitoring it for new content. Historical content from the past 14 days is automatically backfilled, once per source (on a disabled feed, when you enable it).

Item​

An item represents a single article in your feed. When the system discovers and processes an article from a source, it becomes an item in your feed.

An item says:

  • What it is: the title, the description, the URL and when it was published. Every URL is an http(s) URL; a link that is not is left out. publishedAt is the page's publish date, or the date the article states when it gives a time zone; a stated time without one is not used, and publishedAt is then omitted.
  • Whose content it is: the publisher (a website's name and domain, or a social account's name and handle), the author (the byline as written; "Staff" is kept) and, only for a syndicated article or a retweet, the original publisher. The site name comes from the page's own site name when it has one, else from the publisher name we extracted. A news item has no publisher role: the publisher is the source your feed crawls.
  • Where it is about: about.places, from the geocoder's place hierarchy, most central first. Each place has a level (from country down to neighborhood), a short displayText such as Sterling Heights, MI, and a geo with its bounding box and the centre of the box. A place whose hierarchy we have not filled in yet is left out. It appears on the item when we fill it in, and the next time we process the item a news.item.updated announces it as about, even when nothing else changed.
  • Its images: media. We host a copy of each image and give you a signed link that stops working at expiresAt. A website article has at most one image (its main image); a social post has up to 6 (the slides of a carousel, or the cover of a video); a tweet has none. Images smaller than 200 px on the short edge are not used. Do not store the link. If you keep an image, copy the file. Every image has usage: "display_with_attribution": show the source beside it, with the credit when there is one. An image is moderated before it is served, and one that fails is not served. Images stay available for 30 days after the item enters the feed (change it with config.media.retentionDaysAfterAdded, 1 to 365; once set it can be changed but not removed, so set it to 30 to go back to the default). After that the item shows no images and no webhook is sent. media.cover is null when the main image cannot be served: show no image then rather than promoting another one, and read it null-safely (media.cover?.url).

An item is one article or one post. If the publisher edits an article, the same item changes in place: it keeps its id, and you receive a news.item.updated instead of a second copy arriving as a new item. An item can also leave a feed: it then disappears from the list, and only the news.item.removed webhook tells you. See Which articles a feed carries and News feed changes.

updatedAt moves only when your webhooks are owed an added or updated for the item. To poll instead of using webhooks, ask the list for updatedSince with the time of your last poll, a few seconds earlier, and de-duplicate by id and updatedAt. A re-signed image link, or an image leaving its window, is not a change. Polling does not report items that left the feed: only the news.item.removed webhook does.

How It Works​

  1. Create a feed - Your container for content (it starts disabled)
  2. Add sources - Define where to collect articles from
  3. Add filters - Decide which articles it carries, then enable the feed
  4. Automatic monitoring - Sources are continuously checked for new content
  5. Content extraction - Articles are processed to extract structured data
  6. Items appear - Articles that pass the feed's filters become available through the API

Which Articles a Feed Carries​

Each feed decides for itself which articles it carries, and you set it up through the API. Every article is judged by processors, each of which answers a question about it, and include and exclude filters on the feed are written on those answers. An article is published when no exclude filter matches and at least one include filter does, so each include you add lets more articles in. A new news feed has no filters. Enabling a feed through PATCH needs at least one include filter, and is refused with 422 without one. A change to a feed's filters or processors applies going forward only: to new articles, and to articles already in the feed when they next change.

Content types​

Every article and post is labelled with how much it belongs to each of 14 content types:

  • editorial_news: reporting on events and developments
  • opinion_editorial: opinion pieces, columns and editorials
  • life_milestone: a birth, wedding, graduation, a death and the like
  • professional_update: a new job, a launch, a result at work
  • personal_opinion: a person's own view or take
  • celebrity_interaction: a public figure meeting or replying to someone
  • behind_the_scenes: a look behind a production, show or team
  • personal_interest: hobbies, travel, food and other interests
  • daily_life: everyday moments
  • fan_directed: content made for fans or followers
  • sponsored_or_advertorial: paid or sponsored content
  • self_promotion: an account promoting itself or its own work
  • repost: content shared from someone else
  • other: anything that fits none of these

For each type there are two answers: how confident we are that the type is present (0 to 1; the type is present at 0.5 or more), and a significance from 0 to 10. Significance is 0 when the type is absent. A feed's filters read significance, so "at least 5" means the type is present at significance 5 or more. Every type is available to every feed, as outputs of the news_content_type built-in processor (<type>.significance and <type>.confidence). The answers are read by filters only and are not part of the item.

Ready-made sets​

Two sets of our opt-in filters set up a feed in one request (POST /feeds/news/{feedId}/filter-sets/{setKey}):

  • Standard news (standard_news): includes editorial_news or opinion_editorial at significance 6 or more; excludes sponsored_or_advertorial and self_promotion at significance 3 or more; every other type stays out.
  • Creator social (creator_social): includes behind_the_scenes, daily_life, life_milestone, personal_interest, professional_update, fan_directed and celebrity_interaction at significance 5 or more; excludes sponsored_or_advertorial and self_promotion at significance 3 or more.

Both includes (news and creator_posts) optionally take an area and a relevance processor: with an area, only articles in it; with relevance, only relevant ones. The cut-offs follow our latest definition and may be adjusted. To use your own cut-off, remove the opt-in and write your own filter.

Pages behind a paywall that show only a teaser are not added to any feed.

Safety​

A safety floor applies to every news feed and cannot be switched off, and there is no per-feed setting for it. It checks the article's text (its title and body) and excludes articles flagged as unsafe. Reporting on violence (crime, courts, conflict, accidents) is news, so an article is not excluded for violence. It also checks each of the article's images: an image that fails is not served, and the article stays in its feeds without it. Some categories, in the text or in an image, lock an article: it stays out of every news feed until our team reviews it and clears the lock.

Areas​

A feed can be limited to an area: a list of places, at any level, such as a city, a county or a state. You create it as an area match processor and, optionally, an area relevance processor (see Areas), and pass them to the news or creator_posts opt-in, or read them in your own filter. By area match, an article is in the area when a place it is about lies inside the area. A story about a whole state or country does not match a local feed by place; it reaches the feed only through area relevance, which judges from the article's text whether the story is about the region.

When an article leaves a feed​

An article leaves a feed when it no longer passes the feed's filters or the safety floor excludes it. You then receive news.item.removed for it, naming the feed item id you received in news.item.added (data.id). If you never received news.item.added for an article, you never receive a removal for it either. When a feed is deleted, you receive a removal for each article you were sent news.item.added for, about 10 to 30 minutes after the deletion (see Event Delivery: Feed deletion).

Source Behavior​

  • A website can be on several feeds; removing it from one feed leaves it on the others
  • Multiple sources can be added to one feed
  • Removing or disabling a source stops new content from that source reaching the feed
  • Existing items from a removed or disabled source remain in the feed
  • Subscribers to a feed can list its sources but not change them

See Sources for the rules and every error.

Backfilling​

When you add a source, the system automatically retrieves and processes the site's articles from the past 14 days. A disabled feed takes no articles in, so a new feed (which starts disabled) receives them when you enable it: enabling it adds, from each enabled source, the articles the source brought in during the past 14 days that pass its filters. On an enabled feed, a new source's recent articles are added right away, as are those of a source you enable that has not brought them in yet. Either way your feed starts with recent content rather than empty. Each source brings them in once: re-enabling a feed you disabled does not add the articles published while it was disabled, though a source added while it was disabled is brought in.