Tool exports Substack posts as clean Markdown for RAG pipelines without a browser
A developer ran into trouble feeding Substack newsletters into LLMs and RAG systems, since post HTML carries subscribe and share widgets, the archive endpoint no longer accepts offset pagination, and post bodies require a separate request each. To address this, the author built and published a Substack scraper on Apify that uses plain HTTP calls to Substack's public JSON API instead of a headless browser. The Actor outputs each post as self-contained Markdown with the title as an H1, removing widgets while keeping headings, lists, code blocks, captions and footnotes. Incremental runs return only newly published posts, and paywalled content is skipped or flagged by default. Pricing is pay-per-post at $1.20 per 1,000 delivered posts, and the author notes that post content belongs to its authors and must be used within their terms and copyright.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in