Python script or Firecrawl for llms.txt in 2026: the author's pick and why

·9 views

The author recommends a small Python script run in CI for repo-hosted docs, which took about 47 minutes to build and costs nothing per deploy. Firecrawl's hosted generator is better when the source isn't under your control or when an llms-full.txt is needed from a live site, though its auto-written descriptions usually need rewriting. For very small sites, manually writing the file can be quicker than either option. The author also built a free, browser-based llms.txt generator after finding that other tools either routed sites through third-party servers or output bare links without descriptions. The piece walks through the llms.txt spec, including the required H1, optional blockquote, H2 link sections, and an Optional section for skippable content, and it describes how the author's 38-page documentation set was structured and generated.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Python script to check keyword density and placement in HTML articles

A DEV Community article presents a small Python script for content teams that checks keyword density and where a target keyword appears in an HTML page, without needing an SEO plugin. The author argues that keyword placement matters more than raw occurrence counts, so the script prioritizes checks on the title, URL slug, and opening paragraph. Placement is grouped into required, recommended, and natural tiers, which the script reports as errors or suggestions. Because Google publishes no official density target, the script treats density above about 3% only as a warning flag. The code handles Vietnamese diacritics by normalizing text for URL matching and uses BeautifulSoup to parse page elements.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Fair Benchmarks for Large CSV Viewers and Editors: Define Tasks Before Timing

Benchmarking large CSV tools becomes misleading when tools are timed on different operations, such as previewing, full parsing, indexing, filtering, or writing an edited copy. The article recommends defining each task and its start and end points before measuring, and reporting time-to-visible-results separately from complete-scan timings. It provides a synthetic CSV generator and a Python verification script whose expected match count is known in advance. The fixture includes commas, quotes, and embedded newlines, since a physical line count may not equal the CSV record count. The author advises running a small correctness check first, such as 10,000 records, before scaling up.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Example API Responses Need Surrounding Context to Be Useful

A JSON example can look clear while leaving readers unsure about the original request, optional fields, and whether results are complete. The article recommends showing the relevant input, schema or version, field meanings, and how empty or partial results behave. It also advises stating whether an example is synthetic or observed, and keeping credentials and private records out of it. A sample response only demonstrates a contract when its scope is clearly defined. The author notes that a parseable response envelope should not be taken as proof that every item succeeded, so consumers should check the field that determines usable output.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Pending identity verification can resume after an app restart, per a SQLite fixture

A hosted identity onboarding flow can lose track of a pending verification session if a deploy, crash or reschedule occurs before the provider's callback arrives. The article presents a synthetic SQLite-backed fixture in which a second process reopening the same database file sees the existing pending session, its deadline and grant count, with no duplicate session created. Duplicate callbacks are deduplicated by event ID, and a timeout recovery path handles sessions where no callback arrives. The example is local and synthetic only, does not verify real signatures or grant real permissions, and does not change live permissions. The author frames the key question as whether durable storage, not process memory, owns pending state and is read before the restarted process acts.

0 comments Read more at DEV Community