Team cuts Airflow hosting costs to under $150/month by ditching Cloud Composer for GCP VM

·4 views

A machine learning team migrated their self-hosted Apache Airflow and configuration management system from on-premises Linux servers to Google Cloud Platform to reduce maintenance overhead. They initially evaluated Google's managed Cloud Composer 3 service but found it cost approximately $350 per month, nearly double their target budget. Instead, they deployed a self-managed Airflow 3.x instance on a single Compute Engine VM backed by a Postgres metadata database, bringing estimated monthly costs below $150. The architecture also includes a lightweight Flask-based configuration app running on Cloud Run, with configs stored in Google Cloud Storage and ML workloads executed via Vertex AI. The trade-off is that the team must handle Airflow upgrades and infrastructure patching themselves, unlike the fully managed Composer offering.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Building a Snowflake AI Agent Revealed Flaws in Both the Code and Its Evaluation

A developer working with a Snowflake Cortex Agent discovered that poor evaluation scores stemmed not only from agent errors but also from flawed tests and misleading metrics. An object the agent failed to find actually existed in the metadata, but required fixes at two levels: updating the agent's fallback instructions and clarifying the semantic view's SQL-generation guidance. Further investigation revealed that a missing tool-call counter was defaulting to zero, making absent telemetry appear as measured inactivity rather than a data gap. The developer maintained three parallel concerns simultaneously — the agent itself, the evidence of its behavior, and the tests used to judge it. The experience highlighted that misdiagnosing whether a failure lies in the data, the tool, or the agent's prompting leads to entirely different and potentially incorrect fixes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Async Job Queues and Stateful UX Can Fix Broken Social Media Import Pipelines

A development team lost 40,000 Instagram posts after a silent rate-limit error caused their synchronous import pipeline to fail without alerting users. The core problem is that most content import features rely on direct HTTP requests, which are vulnerable to timeouts, API throttling, and partial failures that leave databases with corrupted or incomplete records. The proposed fix involves decoupling the browser session from data ingestion by treating every import as an asynchronous, resumable background task managed by a job queue. A persistent UI drawer subscribes to real-time status updates via WebSockets or Redis-backed polling, so progress is preserved even if the network drops or an external API throttles requests. When slowdowns occur, users are shown an informative banner with an estimated completion time rather than a frozen spinner or a blank screen.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Series Breaks Down How CPUs Work, From Registers to Assembly

A developer named Wesley Bertipaglia has published the fourth installment of a nine-part series on computers. The post focuses on how central processing units (CPUs) function at a fundamental level. Topics covered include instruction sets, registers, the arithmetic logic unit (ALU), the control unit, and the fetch-decode-execute cycle. The article also provides an introductory look at assembly language to illustrate how processors run code. The full post is available on Bertipaglia's personal blog.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Tool Detects Duplicate Code by Running Functions, Not Reading Them

A developer tool called 'assay' identifies duplicate functions by executing them against a fixed set of inputs and comparing output vectors, rather than scanning source text or names. Two functions are flagged as potential duplicates only when their outcome sequences match exactly, turning discovery into a hash-bucket operation instead of a manual or quadratic comparison. The approach uncovered a real-world pair — 'is_wordy' and '_word' — that no name-based or textual tool would have linked, differing by just one character predicate. To avoid false positives, the tool guards against trivially agreeing functions, such as those that always raise the same error, return a constant, or merely copy their input. The tool is available for both Python and JavaScript via 'pip install assay-checks' and 'npm install -g assay-checks' respectively.

0 comments Read more at DEV Community