SShortSingh.
Back to feed

Dagster and dbt: Do You Need Both?

0
·1 views

A common question is whether teams using Dagster also need dbt, since both draw lineage graphs, schedule work, and run tests. The author explains they are different kinds of tools: dbt is a SQL transformation framework that orders and tests SQL models but does not schedule itself, while Dagster is an orchestrator that handles scheduling, retries, partitions, and arbitrary Python. The overlap lies in lineage and testing, but Dagster's graph covers the whole pipeline, including Python loading and model training. dbt remains valuable when transformations are mainly SQL in one warehouse, when analysts write models, or when portability across orchestrators matters. The author's own production pipeline runs on Dagster without dbt, which suits teams where transformations are mostly Python and one engineer manages everything.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Sloth Sort: local Gemma 3 tool automatically renames messy screenshots and downloads

Sloth Sort is an open-source tool that watches the Desktop and Downloads folders for files with generic names like screenshots and scans. When such a file appears, Gemma 3 running locally through Ollama identifies its contents and renames it with a date-first, descriptive name. Only throwaway names are changed, and every rename can be reversed from a review page on the local machine. The model runs entirely on the user's computer so sensitive screenshots never leave it, and it unloads after five idle minutes. The project was submitted to the Hacktoberfest Open-Source AI Challenge.

0
ProgrammingDEV Community ·

Asynchronous JavaScript: How Non-Blocking Operations Work

Some JavaScript tasks, such as fetching data from an API or waiting on a timer, take time to finish. Asynchronous JavaScript lets the language handle these tasks without halting other code from running. Synchronous code executes line by line in order, while asynchronous code can schedule work, like a setTimeout callback, to run later. The article's example shows a log statement printing before a delayed callback, illustrating this behavior. Common uses include fetching API data, loading products on shopping sites, sending login details, uploading files, and communicating with databases through a backend.

0
ProgrammingDEV Community ·

ShortSingh: Android app's model search now hides models your phone cannot run

Off Grid AI Mobile (OGAM) has added a filter to its Android model search so users avoid downloading AI models their phones cannot load. Results are filtered by the phone's RAM, showing only models that will load and marking those that only just fit as Tight. Each model appears once even when many uploads exist, and models needing a newer engine, speed-up helper files and image generators are left out of chat search. The changes arrived in versions 0.0.116 and 0.0.117, and search is free with no account required. Once downloaded, models run on the phone without an internet connection.

0
ProgrammingDEV Community ·

Arrange My Files AI Builds Reviewable Plans to Organize Documents Without Touching Originals

Arrange My Files AI is a tool that turns cluttered folders into organization plans users can review before anything is copied. Original files stay in place, and the included batch is copied into a new output session. Files with clear content go into category folders, while uncertain or unsupported files are placed unchanged in a Needs Review folder. The author tested the app with nine dummy files, where the engine renamed some files based on document titles while preserving descriptive names. The developer notes that the system performs quick, bounded analysis rather than full document understanding, so some files still require manual review.

Dagster and dbt: Do You Need Both? · ShortSingh