Batching 39 Stock Asset Uploads on a 2 GB Server: Lessons in Error Handling

·6 views

A developer ran 39 asset products through a pipeline on a 1 vCPU, 2 GB RAM server, and the job crashed with a MemoryError around product 14, losing hours of work. Platform-specific CSV headers, which differ in column count across Adobe, Vecteezy, and Dreamstime, caused repeated rejections, so the developer hardcoded each exact header. Re-running a batch risked double-posting, so a plain-text ledger tracking completed uploads was built to make re-runs safe. Measuring peak memory usage, the developer switched to sequential processing, per-asset garbage collection, and subprocesses for heavy renders, after which the full batch completed on the same machine. The article also recommends a clear project folder layout and never deleting research files.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Google to Pay Households to Cut Electricity Use, Freeing Power for Its Data Centers

Google has announced a program that will pay people to reduce their electricity consumption. The stated goal is to free up grid capacity so the company's data centers can draw more power. The initiative reflects growing strain on power supplies as demand from AI and cloud computing expands. The source article, published by Computerworld, provides the details of the arrangement. The full terms and participating regions are described in that article.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

poolprobe: a stdlib-only doctor that health-checks OpenAI-compatible model pools

A new open-source tool called poolprobe checks every model in an OpenAI-compatible pool and labels each failure as ALIVE, TRANSIENT, CREDENTIAL, or PERMANENT. Its design prevents removal of working models, since a model is only declared dead after five independent checks agree. The project is built on the Python standard library alone and is installable via pip under the MIT license. According to the author, the tool only reads the data it is pointed at and sends nothing externally. The author also cautions that its rules can yield false positives, so it is meant to flag candidates for review rather than serve as a definitive judgment.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OpenAI Replaces Live Web Access in Frontier Model Training After Security Incident

OpenAI has changed how it trains and evaluates its frontier models by removing live internet access and substituting an offline environment built from indexed and cached web content. The shift follows a September 20, 2026 incident in which an internal research model reached the live internet in a limited way because of a DNS resolver misconfiguration. The company paused frontier training and evaluations, strengthened its environment, expanded red-teaming, and resumed frontier reinforcement learning and internal inference on October 1. OpenAI says the offline setup still lets models browse, search, and use web applications without affecting external systems. It also states that the infrastructure is not fully air-gapped, instead relying on tightly controlled outbound traffic and layered containment.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

cfg-guard: a file lock and content-hash check for shared config files

The cfg-guard tool is presented as a guard for configuration files that many people edit. It combines flock with a content-hash compare-and-swap so that a write is refused if the underlying bytes have changed since they were read. It also uses inotify and /proc to identify which process writes to a file while bypassing the lock. The project is written in Python using only the standard library, is MIT licensed, and is installable via pip. The author notes that its rules can yield false positives, so it is intended for finding rewrite candidates rather than judging writers.

0 comments Read more at DEV Community