Why GitHub Blocks Search Results Beyond Page 10: The Deep Pagination Problem

·1 views

GitHub's Search API deliberately returns a 422 error when users request results beyond the first 1,000 entries, blocking access to page 11 and beyond with 100 results per page. The restriction exists because of a fundamental database limitation: SQL's OFFSET clause cannot jump directly to a position in a B-tree index and must scan and discard every preceding row one by one. This means fetching page 50,000 of a 20-results-per-page query forces the database to walk through nearly a million rows before returning any data, making query cost proportional to the offset rather than the page size. A real-world test on a 5-million-row PostgreSQL table confirmed the issue, with an OFFSET of 999,980 causing the engine to scan exactly one million rows before delivering just 20 results. GitHub's hard cap is a deliberate safeguard against this performance problem, which can silently affect any application using offset-based pagination on large datasets.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

XML Tags in AI Prompts Reduce Ambiguity and Improve Output Consistency

Developers working with large language models like Claude and GPT often encounter unreliable outputs when prompts contain multiple instructions, context, and examples together. Wrapping distinct prompt sections in XML tags — such as <instruction>, <context>, and <example> — creates clear boundaries that models can parse accurately. LLMs are well-suited to recognise XML structure because such markup is heavily represented in their training data, and Anthropic explicitly recommends this approach for Claude. The technique also extends to model responses, where asking the model to return output inside named tags makes programmatic extraction straightforward. XML tagging is most beneficial for complex, multi-part prompts, while simple single-instruction queries gain little from the added structure.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds PostureGuard, a domain security scanner using Next.js and Python

A developer named Sam Dossou built PostureGuard, a web application that scans domains and generates security posture reports covering TLS, HTTP headers, and open ports, scoring them on a 0–100 scale. The app uses a Next.js frontend paired with a Python background worker, with PostgreSQL serving as both the database and the job queue via a SKIP LOCKED query pattern — eliminating the need for a separate message broker. The Python worker checks TLS certificate validity, key HTTP security headers, and exposed sensitive ports using only the standard library to minimize dependencies. To prevent misuse, users must verify domain ownership by adding a DNS TXT token before any scan can run. The project is now live on Azure Container Apps, with authentication handled via bcrypt-hashed passwords and server-side sessions stored in the database.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

DCAB 0.12.0 adds test case for Word comment 'done' state change in XML metadata

The Document Change Assurance Benchmark (DCAB) version 0.12.0 introduces a new deterministic test case called review.modern_comment_done_state_changed, targeting a specific boundary in WordprocessingML document structure. The case examines how a comment's 'done' status can shift from done='0' to done='1' inside word/commentsExtended.xml without altering any visible text, anchors, or other package members. This distinction matters for document review tools because stored metadata changes may not be reflected in a document's visible content or classic comment data. The release passed continuous integration testing across Python 3.11 to 3.13 and includes validation support via an optional DocFence 0.27 adapter. DCAB now contains 23 paired synthetic test cases, with release artifacts and the full corpus available on GitHub and Hugging Face.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds ephemeral encrypted chat where server never stores messages

A developer has released elm.chat, an open-source encrypted chat application built on Cloudflare Durable Objects and Workers, designed so that no message content is ever stored server-side. Each chat room is a temporary space where encryption keys are derived in the browser using HKDF-SHA-256, keeping the server blind to message content. Messages and files are encrypted with AES-GCM-256 before being relayed through a Durable Object, which handles routing but never persists transcripts. When a new participant joins, chat history is synced directly from already-connected peers rather than loaded from a server, meaning if no peer holds a message, it is permanently lost. The project is licensed under AGPL-3.0, though the developer acknowledges that connection metadata such as IP addresses and timing remain visible to Cloudflare.

0 comments Read more at DEV Community