Read replica lag nearly filled Photoroom's primary database disk during a backfill

·1 views

During a large backfill, a lagging read replica at Photoroom caused PostgreSQL to retain write-ahead logs on the primary, growing them from about 5 GB to 31.7 TB. Automatic storage growth had pushed provisioned disk to around 43 TB, closing in on a hard 60 TB limit. The team estimated only five to seven days remained before exhaustion, while the backfill needed several more weeks. They paused the work and enlarged the replica, which let the logs drain and the database return to normal. The team now monitors retained WAL alongside replica lag and limits backfill concurrency and transaction size.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Phantom AI: A Halloween Chatbot Running on Local Open-Weight Models

A developer has built Phantom AI, a spooky interactive chatbot themed around the Ghost in the Machine concept for a Hacktoberfest challenge. Users can converse with different ghost personalities, ask questions, and generate eerie stories through a haunted terminal-style interface. The front end is built with HTML, CSS, and JavaScript, while responses come from the Qwen3 open-weight model run locally through Ollama. The project aims to show how open-weight models can power creative applications with more user control over data and experimentation. The author notes trade-offs such as hardware requirements and storage needs, and the public website may not fully work without a local Ollama setup.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Crystal Explained: Ruby-Style Syntax With Compiled Native Speed

Crystal, a language first released in 2014, aims to keep Ruby's readable syntax while compiling programs to native machine code. It is statically typed, with types largely inferred, so developers get compile-time checks without heavy annotation. The language offers compile-time macros for code generation and produces single binaries through LLVM, requiring no virtual machine or runtime. Its main drawbacks are a small ecosystem, fewer libraries and jobs, maturing tooling, and historically weaker Windows support.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OpenAI DALL-E vs DeepAI for Small Catalog Apps: Storage and Cost Trade-offs

For a small B2B catalog application, the article recommends OpenAI's image generation API as the safer default when output quality and room to grow matter more than the lowest starting cost. DeepAI is presented as a viable option only for narrow, latency-tolerant workflows that pass a representative acceptance test, and Stability AI as a credible third choice when control over the model matters. The author argues that the image endpoint is only part of the system, since generated files need private storage, safe retries, and metadata linking them to their source descriptions. Budget should be judged by acceptance rate, end-to-end latency including retries, and how often a second prompt is needed, rather than by a price sheet alone. The piece also outlines a storage pattern using deterministic private object keys tied to catalog item and prompt revision, so duplicate queue deliveries do not create duplicate objects.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

HauntShell: A Haunted Terminal and Séance Altar Honoring Dead Technologies

HauntShell is an open-source retro terminal built for the Hacktoberfest AI Challenge's Ghost in the Machine theme, timed to Halloween and Día de los Muertos. It intercepts common shell commands such as git commit, rm -rf, and sudo reboot, responding with eerie, darkly humorous commentary and tracking invented "ectoplasm levels." A separate digital altar lets users converse with personas representing obsolete technologies, including Adobe Flash, Ada Lovelace, COBOL, and GeoCities. The project runs on open-weight models through Ollama, with cloud backups via Groq and Gemini and a built-in offline fallback for testing. Its ambient soundscape is generated entirely in the browser using the Web Audio API, requiring no sound files or internet connection.

0 comments Read more at DEV Community