How LectuLibre Built a Chunking System to Translate Full Books via Claude API

·3 views

AI book translation platform LectuLibre discovered that the primary engineering challenge was not translation quality but Claude API's output token limits, which cap responses at around 4,096 tokens — far short of a full book's 120,000 tokens. To work around this, the team built a paragraph- and sentence-aware text chunking system that splits source content into manageable segments of up to 8,000 tokens each. An overlap of roughly 200 tokens between consecutive chunks preserves narrative context and reduces translation errors at boundaries. The system uses the tiktoken library for token counting and supports parallel processing of chunks to minimize user wait times. The approach also handles common PDF and EPUB extraction issues such as headers, footers, and chapter structures.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds AgentGuard CI tool after testing 12 attack payloads on own repo

A developer created AgentGuard, an open-source CI security tool designed to detect malicious instructions hidden inside AI coding agent configuration files such as AGENTS.md, MCP server definitions, and hooks. To validate the tool, the author planted 12 distinct attack payloads in their own repository, including hidden instruction overrides, zero-width characters, typosquatted server hosts, and committed API keys. AgentGuard detected all 12 threats deterministically without using an LLM in the scan path, and produced zero false positives on a clean control repository. A scan of 30 public repositories on GitHub, including those from Google, Microsoft, and Nextcloud, uncovered five with critical findings. The tool is MIT-licensed, free for public repositories on the GitHub Marketplace, and its own CI gate blocked the developer's pull request during a self-test, confirming it works as designed.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Qwen Releases Qwen3.8-Flash-Next, an Experimental 125B AI Model for Coding and Agents

Qwen has published Qwen3.8-Flash-Next, an open-weight experimental language model with a vision encoder, available on Hugging Face in Transformers format. The model features 125 billion total parameters with 6 billion activated, supports a native context window of 262,144 tokens extendable to one million, and is designed for coding agents, multilingual software engineering, tool use, and multimodal computer tasks. It is intended as an architecture preview underpinning the upcoming Qwen4 release, and is distinct from the production-hosted Qwen3.8-Flash service, which includes additional features. The model achieves notable benchmark scores, including 81.0 on SWE-bench Multilingual, 91.9 on LiveCodeBench v6, and 84.5 on AndroidWorld, outperforming several earlier Qwen models on coding tasks. Its license is listed as 'other' and does not explicitly grant commercial-use rights, meaning legal review is recommended before any commercial deployment.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

React Context Is a Dependency Injection Tool, Not a State Manager

Many React development teams misuse the built-in Context API as a general-purpose state management solution, which can severely degrade application performance. Because React re-renders every component consuming a context whenever its value changes, high-frequency updates — such as keystrokes — can trigger costly cascades across the entire component tree. An audit of an enterprise dashboard found that form input latency reached 250ms per keystroke due to this misuse; migrating to Zustand reduced that figure to just 12ms. Zustand and similar libraries use selector-based subscriptions, ensuring only the directly affected component re-renders on each state change. Experts recommend reserving Context for low-frequency global data like themes or authentication status, and using external stores for any state that updates rapidly.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Firefox Extension Exports PDFs Using Only Browser APIs, No External Library

A developer building Longshot, a Firefox screenshot extension, created a 336-line PDF exporter without relying on any external PDF library. The approach works because two key tasks a PDF library would normally handle — JPEG encoding and zlib compression — are already natively supported by the browser through canvas.toBlob and CompressionStream respectively. These browser APIs align directly with PDF's built-in /DCTDecode and /FlateDecode stream filters, allowing image data to pass through without re-encoding. The developer notes this is a narrow solution covering only image placement and invisible text layers, not the full range of features a general PDF library provides. A key engineering decision was to assemble the file as raw bytes with precisely measured offsets, since a single incorrect entry in PDF's cross-reference table renders the entire file unreadable without any obvious error.

0 comments Read more at DEV Community