How LectuLibre Built a Chunking Pipeline to Translate Full Books Using AI

Startup LectuLibre developed an AI-powered book translation pipeline after discovering that sending entire books to large language models in one request was too costly, unreliable, and prone to quality degradation over long contexts. The team split books into overlapping token-aware chunks of roughly 3,000 tokens each, using tiktoken for precise token counting after finding LangChain's character-based splitter caused chunks to exceed model limits. Each chunk is translated using Anthropic's Claude 3.5 Sonnet alongside a dynamic context buffer containing a running glossary of character names and terms, plus a rolling summary of previously translated chapters stored in PostgreSQL. After translating each chunk, the pipeline extracts new terminology and generates a short summary to carry forward into the next chunk, preserving stylistic and narrative consistency throughout. For a typical 120,000-word novel, this approach produces around 60 chunks, with the assembled output forming a complete, context-coherent translated book.