GCP us-west1 Outage: How a Fiber Maintenance Job Took Down 20+ Services

·1 views

On August 20, 2026, a planned fiber-optic maintenance operation inside Google Cloud's us-west1 region unexpectedly reduced inter-campus network capacity, triggering a two-hour, twenty-two-minute outage. Automated traffic rerouting systems failed to compensate for the capacity loss, causing congestion that cascaded into the region's core infrastructure. The disruption reached critical shared components — including Spanner Paxos consensus and the Unified Metadata Server — which then propagated failures across more than two dozen Google Cloud products, spanning compute, storage, databases, and identity services. Google's incident report highlights that the outage's broad blast radius was shaped by shared underlying infrastructure rather than any direct fault in individual products. The incident underscores that cloud failure domains are defined by shared physical and logical infrastructure, not by the list of services a customer happens to be running.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Apple Faces Backlash Over Persistent In-System Ads Added to iOS

Apple has introduced what users are describing as persistent advertisements within iOS, prompting widespread frustration among iPhone users. The promotions appear to be built into the operating system itself, making them difficult for users to dismiss or avoid. The move has drawn significant criticism from the Apple user community, who feel the ads are intrusive and out of character for a premium platform. Many users have taken to online forums to voice their dissatisfaction with the change. The development raises broader questions about Apple's approach to in-platform monetization on its mobile operating system.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Pre-Launch Checklist: How to Instrument and Test a Support Chatbot Right

Shipping a support chatbot without proper preparation can lead to hard-to-debug failures, since real user behaviour is far messier than controlled test environments. Developers should ensure distributed tracing across all services — including authentication, API gateways, retrieval, and model providers — using shared request and trace IDs. Conversation logs must be handled carefully to avoid storing sensitive customer data such as payment details or personal information unnecessarily. Test sets should be built from real support queries and cover edge cases like misspellings, multi-turn exchanges, out-of-scope requests, and prompt injection attempts. Versioning the evaluation set is essential so that changes to prompts or models can be reliably assessed rather than guessed at.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Client Timeouts Should Trigger In-Doubt Leases, Not Immediate Retries

A software engineer analyzing agent planner logs discovered that a client-side timeout followed by an immediate tool-call replay caused duplicate side effects within the same conversation session. The root issue is that a client timeout is not a confirmed abort — the original call may still complete on the server side, as happened four seconds after the client gave up. To prevent duplicate writes, the engineer proposes that any side-effecting tool result should only be committed when three tokens match: a lease ID, an idempotency key, and a conversation epoch. Under this design, a timeout should place the call in an IN-DOUBT state and trigger a recovery path — commit, compensate, or reject — rather than blindly replaying the payload. The proposal is a design pattern aimed at shared remote executors where slot reuse and late delivery make duplicate execution a structural default, not an edge case.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How AI silently fills code gaps when requirements are ambiguous

When AI tools generate code from vague specifications and limited examples, they quietly make undocumented assumptions to fill logical gaps — without flagging them. Engineers at knopfdruck.ai discovered this after realising their AI had implemented calculation rules that differed from what was actually intended, even though all tests passed. The core problem is that a brief description plus sample data does not define a single rule, but a family of rules that all fit the samples equally well. Their fix was to require the AI to articulate its interpretation in plain prose before writing any code, iterating until the AI's understanding matched the intended logic. Tests are then built from the agreed rule rather than just the original samples, ensuring edge cases are covered and hidden decisions are surfaced early.

0 comments Read more at DEV Community