LinkDigest Tool Extracts Full Content from Xiaohongshu and Similar Walled Platforms

·1 views

A developer has built LinkDigest, a hosted service that converts Xiaohongshu, Douyin, TikTok, YouTube, and X posts into structured text usable by AI language models. Standard web fetches on these platforms return little usable content due to login walls, tokenised share links, and media-heavy formats that leave almost no readable HTML. LinkDigest addresses platform-specific hurdles — such as Xiaohongshu requiring a desktop user agent and a two-step cookie handshake before note content loads — to extract transcripts, image descriptions, OCR text, and metadata as Markdown or JSON. The service is available as an MCP server, a REST API, and a web console, with three free digests requiring no payment details. Limitations are disclosed openly: Bilibili blocks the service's server address, Instagram support is unverified, and Douyin requires logged-in session cookies with no anonymous workaround available.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why AI Coding Tools Need Transparent Pricing and Task-Based Model Selection

A software developer argues that AI coding agents lack the cost transparency and accountability that other trades take for granted, often trapping users in a sunk-cost cycle of iterative prompting without clear delivery guarantees. Unlike traditional software marketplaces where engineers were rated publicly and priced honestly, current AI tools normalize vague estimates and unpredictable outcomes. The author also raises concerns about widening access inequality, noting that frontier model hardware grows more expensive rather than more affordable over time. To address this, the developer proposes an 'intelligence ladder' — a framework that maps tasks to specific model requirements and cost estimates before work begins. After a year of development, a working version of this system is now ready to be shared publicly.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

pdlc-skills Adds Three Tools to Track Project Status, Change Impact, and Trends

The pdlc development lifecycle framework offers three visibility tools — a statusline, /pdlc-status, and /pdlc-retro — designed to monitor software projects across different timescales. The statusline displays a feature's current stage, next step, run mode, test results, and time spent in the current stage in a single compact line. The /pdlc-status command verifies project data against changelogs and git tags, flagging features that have stalled without shipping. A relation-mapping tool, /pdlc-relate, automatically identifies how features connect through six relationship types by scanning requirements text for dependency language. All three tools read from shared state files under docs/.pdlc-state/, meaning the accuracy of their output depends entirely on how well those files are maintained.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Better AI Answers Come From Cutting Context, Not Adding More

A software developer argues that reducing what goes into an LLM's context window typically improves output quality more than expanding it. Because language models attend to all provided tokens equally, irrelevant material competes with useful information and can anchor the model on wrong details. Content buried in the middle of a long context also receives less model attention than material placed at the start or end, meaning placement matters as much as inclusion. The author recommends a workflow of generous retrieval followed by aggressive trimming — removing full files, stale conversation history, and boilerplate — keeping only the task, the directly relevant facts, and the desired output format. This approach simultaneously improves answer accuracy and lowers token costs, making it one of the few optimizations that benefits both quality and budget.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Running Database Auth in Next.js Middleware Hurts App Performance

A technical analysis warns that placing database-backed authentication checks inside Next.js Middleware is a common architectural mistake that degrades application performance at scale. The Next.js Edge Runtime, built on V8 isolates, lacks support for native Node.js modules that most database drivers depend on, making direct database connections impossible without clunky workarounds. Because middleware executes on every matched request — including page navigations, server component fetches, and background revalidations — firing a database query each time can overwhelm and exhaust database connections. Developers are advised to adopt a two-layer authentication approach: using lightweight JWT signature verification via the jose library at the edge, while deferring role checks and session validation to Server Components or Route Handlers running in the full Node.js runtime. This separation preserves both security and performance without overloading the database.

0 comments Read more at DEV Community