DIY guide · Medium API
One pipeline. Every Medium source you care about.
Niche newsletters, competitor monitors, and research products all run on the same insight: Medium is a firehose of signal if you can normalize it. This guide shows how to build a single ingestion layer—not five brittle one-offs.
Why aggregators fail without an API layer
Most "aggregators" die in maintenance. Someone scripts three publications, hard-codes CSS selectors, and ships. A month later one publication redesigns and the feed is empty. Meanwhile a competitor added twelve new sources you never got to.
The fix is not more scrapers. It is a single schema for articles—id, title, url, preview, published_at, source—and a pipeline that fills it from many Medium surfaces.
Who pays for this (and why)
Newsletter operators want tomorrow's digest built tonight. Media startups want topic pages that update without a newsroom. B2B tools want "everything written about {{keyword}}" for sales intelligence. SEO plays want programmatic pages that are actually useful because the underlying data is fresh.
All of them share one requirement: reliable metadata at scale.
Designing the pipeline
Think in layers, not endpoints:
Sources are the entities you follow—publications, writers, tags, saved searches. Ingestion is the scheduled job that asks "what's new?" for each source. Normalization maps every response into one internal Article model. Distribution is whatever your customers see: RSS, GraphQL, email, or a web UI.
Zenndra gives you the ingestion verbs: publication feeds, user article lists, tag top feeds, and keyword search. You bring the schedule and the database.
Making it production-grade
Production means deduplication on article_id, cursor or timestamp tracking so you only process new items, and backoff when you hit rate limits. It means alerting when a source returns zero items three runs in a row—often a signal to fix a slug, not to panic.
It also means being honest about attribution. Aggregated products that link out clearly and cache responsibly last longer than those that strip branding.
How Zenndra changes the math
Instead of N scrapers for N sources, you hold one API key and one response shape. Your engineers ship features—filters, alerts, digests—not parser patches. When Medium moves a div, Zenndra absorbs the change on the infrastructure side.
That is the difference between a weekend project and something you can sell.
A realistic first version
Week one: three publications and one writer, cron every six hours, store title/url/id. Week two: add search for two keywords, merge into the same table. Week three: a simple UI or email template. You do not need forty sources on day one—you need proof the pipeline never silently stops.
Expanding without rewrites
Add sources by adding rows in your config, not by forking code. Tag feeds for trending topics, user endpoints for influencer columns, search for long-tail pages. The aggregator becomes a strategy layer on top of a stable transport layer—and that is when it becomes a business.
Starter code
// Nightly job: merge publication + search into one table
const sources = [
{ type: 'publication', id: '7f60cf5620c9' },
{ type: 'search', query: 'machine learning' },
];
// Normalize each item to { article_id, title, url, source, fetched_at }
// Upsert on article_id — skip duplicates