SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Adding llms.txt, llms-full.txt and tdmrep.json to a Next.js App Router Site

0
·3 views

A tutorial explains how developers can add machine-readable files that help AI coding assistants and AI search engines understand their sites within about ten minutes. It outlines the required structure of an llms.txt file under the llmstxt.org specification, including a single top-level heading, a short summary blockquote, and descriptive link lists. The piece also covers common robots.txt mistakes under RFC 9309, such as unintended prefix matches that can block or leak paths for AI crawlers. It demonstrates serving /llms.txt through a static Next.js App Router route handler with caching headers. The article additionally touches on serving a W3C tdmrep.json file and on tooling for converting existing documentation into these formats.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Seccomp Filters Should Validate Architecture Before Checking Syscall Numbers

A seccomp policy can appear complete yet leave an alternate syscall path open, because filters see only a syscall number and architecture identifier rather than semantic function names. The same integer can mean different things under different ABIs, so a policy checking only the syscall number can be bypassed. Architecture validation should therefore occur at the start of the filter, as shown in Linux seccomp(2) documentation, with unknown architectures rejected. The article recommends choosing default actions matching the risk level, maintaining separate filters and tests for each ABI, and failing closed when instructions cannot be classified. Seccomp is also limited because it cannot inspect pointer contents, so it is usually paired with namespaces, file permissions, and capabilities.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Perfect Annotation Agreement Can Hide Sparse Labelling Coverage

A single 100% agreement figure can make two annotation files appear more complete than they are. In the bundled partial example, only one of twelve records is labelled by both sides, so conditional agreement reads 100% while both-labelled coverage is just 1/12. The metric counts only records labelled by both sides, so it ignores the remaining records entirely. The article's review tool therefore checks that both checkpoints share the same records, taxonomy, and order before comparing them, and it keeps manual decisions separate from the original annotations. It also stresses that agreement is not accuracy and that the two annotation roles are not verified as independent people.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Guide Outlines C# Techniques for High-Throughput RAG Pipelines and Vector Database Tuning

A DEV Community article presents an architectural blueprint for building high-throughput agentic and retrieval-augmented generation pipelines in C#. It covers vector database tuning, including IVF cluster parameters such as nlist and nprobe, alongside HNSW indexing. The piece explains how Span, ReadOnlySpan, and pooled memory buffers can reduce garbage collection overhead during heavy vector operations. It also includes a SIMD-accelerated cosine similarity example and describes a two-tier L1/L2 caching model to limit redundant similarity queries. The author reports benchmark results indicating a reported throughput gain of about 4.8x over default settings when HNSW tuning and caching are combined.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds Neural Network Basics in Plain JavaScript to Understand How AI Learns

A JavaScript developer set out to understand how AI models learn after realising that using AI tools was not enough to explain their inner workings. Rather than relying on frameworks like TensorFlow or PyTorch, they chose to implement the fundamentals from scratch in plain JavaScript. They began with a simple task of teaching a program to predict the sum of two numbers from examples, illustrating the core idea of supervised learning. The article then explains how a neuron combines weighted inputs with a bias, and how a loss function measures prediction error so the model can adjust itself. The author presents the approach as a step toward eventually building a GPT-style model.

0 comments Read more at DEV Community