Dev Team Builds Tool to Auto-Generate Airflow DAGs from DataHub Metadata

·1 views

Two developers created a tool called datahub-dag-generator during DataHub's Build with DataHub Agent Hackathon that automates the conversion of DataHub metadata into Airflow DAGs. The tool accepts a table name as input, traces the lineage graph backwards to identify all upstream dependencies, and generates corresponding Airflow task code. It operates in two modes: an LLM-powered agent mode using models like Claude via OpenRouter, and a script mode designed for CI pipelines that bypasses the LLM entirely. A key design choice keeps the LLM from writing code directly — it produces only a structured plan, which a separate deterministic renderer then translates into actual task code. The project is still in early stages, lacking dbt and Spark operator support, and has primarily been tested on the DataHub NYC Taxi sample dataset.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Dev Team Documents Automated User Hardening Process for Aerospike Clusters

A standard operating procedure (SOP) has been published outlining how to automate user hardening on Aerospike database clusters using Ansible playbooks via Jenkins and AWX. The process involves removing unwanted users from target clusters across DEV, UAT, and PROD environments. Operators must supply key variables — including cluster name, roles, and username — before launching the job through either AWX or Jenkins. Jenkins triggers the configured AWX or Ansible job template, after which execution status must be verified on both platforms. Prerequisites include configured credentials, a prepared inventory of primary cluster nodes, and validated user-hardening details.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Agent Session Portability Masks Deeper Governance and Data Security Risks

A developer tested session-migrate 0.8.0 by freezing a real Claude Code session and converting it into seven different AI coding agent formats, including Codex, GitHub Copilot CLI, and Mistral Vibe. While conversational context transferred reasonably well, organizational controls such as approval workflows and audit trails did not migrate at all. The experiment also revealed a data-governance risk: session files may contain production logs, schema details, and real data samples, meaning format conversion doubles as an unintended data export route. Tools like Slack Code take the opposite approach by anchoring agent work records within a shared, permission-controlled platform rather than individual developer machines. The author concludes that agent session portability is fundamentally a data-governance challenge, not merely a developer productivity feature.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Five Design-to-Code AI Tools Tested on the Same Outdated SaaS Dashboard

A developer evaluated five design-to-code tools — Flowstep, Anima, v0 by Vercel, Lovable, and Bolt.new — by feeding each the same outdated SaaS analytics dashboard screenshot and an identical redesign prompt. The goal was to assess how well each tool could modernize the interface while preserving core functionality, and to examine the quality of the generated frontend code. Flowstep ranked highest overall for visual design, hierarchy, and code structure, while Lovable also performed strongly with a clean interface and solid code output. Anima, v0, and Bolt.new produced usable results but each required additional refinement in either design or code quality. The test highlighted that design-to-code tools vary significantly in their strengths, with some excelling at visual output and others at developer-ready code.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How AI and Real-Time Data Are Reshaping Hotel Dynamic Pricing Systems

A data engineer with nearly a decade of experience in travel industry pricing systems has detailed the architectural and engineering challenges behind modern AI-driven hotel dynamic pricing. Unlike traditional revenue management tools that updated rates once or twice a day, today's systems require sub-second inference and continuous price adjustments fed by dozens of real-time data sources. Effective feature engineering for these models draws on temporal patterns, competitor rate data, demand signals like search trends and cancellation rates, and guest-level behavioural data from CRM and booking platforms. Additional contextual inputs such as weather forecasts, flight loads, and local events are also evaluated, though distinguishing genuinely predictive signals from noise remains a key challenge. The author notes that many organisations invest heavily in machine learning models but struggle when deploying them within legacy infrastructure that cannot support real-time recalculation.

0 comments Read more at DEV Community