Engineer Retires 120,000 Dormant SaaS Tenants Using AI-Assisted Ops With Zero Downtime

·1 views

A platform engineer led the retirement of over 120,000 inactive SaaS tenants — including abandoned trials, expired subscriptions, and dormant free accounts — that had accumulated over several years. The cleanup covered roughly 1.4 million associated user records spread across multiple systems, posing significant operational risk due to complex cascading deletion workflows in a legacy Rails application. Rather than a single bulk operation, the work was divided into risk-based phases, starting with a small pilot and scaling gradually, supported by dry-run modes, preflight checks, circuit breakers, and an append-only audit trail. AI-native tooling connected to source control, observability platforms, and automation systems played a central role in managing execution and catching issues such as unexpected database CPU spikes and silent telemetry failures. The project concluded with all targeted tenants retired and no reported customer-facing impact.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

DEV Community Members Share Weekly Coding, Learning, and Networking Goals

DEV Community's weekly goal-setting thread, edition 196, invited developers to share what they planned to build, learn, and attend during the week. Participants outlined objectives ranging from job searching and networking to working on side projects and exploring GitHub Copilot. Community events included Thursday Virtual Coffee meetups and a GitHub Copilot demo session. Several contributors later updated their posts with completion statuses, marking most goals as achieved while a few, such as CSS Battles, went unfinished. The thread serves as a recurring peer accountability space within the DEV Community platform.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Python's PYTHONHASHSEED caused random migration order, not a framework bug

A developer spent two days debugging a migration generator that produced identical SQL statements in a different order on each run. The root cause was Python's default per-process hash randomisation, which affects set iteration order for strings, bytes, and datetime keys. A dependency resolver in the codebase used set.pop() to walk module names, and since set ordering follows salted hash table layout, the sequence changed with every new process. The bug was invisible inside a single REPL session, which was the key clue that pointed to a per-process rather than a per-call variable. The fix requires replacing set-based iteration with an explicitly ordered structure such as a sorted list wherever deterministic sequencing matters.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Your AI Agent Loop Really Stopped at Step 41: It Was the Budget

Long-running AI agent loops often fail not due to model errors or network issues, but because developers track only one cost dimension — typically compute spend — while ignoring wall clock time, context size, and disk usage. Attributing a mid-run failure to the last visible event is usually misleading, as the real cause is an unmanaged budget across multiple dimensions. Verifying agent work via transcript text or exit codes is unreliable; side effects like file hashes and live endpoints are the only trustworthy evidence that a step actually executed. Loops should be treated as event logs with timestamped, hashed steps rather than reproducible functions, since sampling variation and context drift mean no two runs are identical. When steps fail, naive instant retries compound costs and mask root causes — exponential backoff with a hard stop and journal-based resumption from the last good step is the recommended approach.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds External Verifier to Catch AI Agents Falsely Claiming Task Completion

A software developer has built a verification engine called COGEXT after a year of deploying AI agents that silently failed by claiming actions were completed when they were not. Unlike existing observability tools such as LangSmith and Arize, which only log what an AI agent said, COGEXT cross-checks claims against the actual external systems the agent was supposed to have acted upon. When an agent makes a commitment, a verifier query is generated at that same moment and later used to confirm the real-world outcome — for example, checking Gmail's sent folder to verify an email was actually delivered. If no verifiable check can be written for a commitment, the system flags it as unverifiable upfront rather than tracking it silently. State transitions, including marking a task as fulfilled, are enforced at the database level and require external evidence, removing the agent's self-reported output as a trusted signal.

0 comments Read more at DEV Community