Hydra ETL 0.20 preserves Snowflake types when loading into PostgreSQL

·2 views

A developer found that moving data from Snowflake to PostgreSQL can silently corrupt values, such as an order ID shown one digit off after passing through a floating-point conversion. To prevent this, the Hydra ETL tool was given a Snowflake source that maps each value and column type exactly, or halts the run before writing anything. In a test with 50,000 orders, row counts, totals and the smallest ID matched on both sides, and the generated PostgreSQL table used bigint, numeric, timestamp and jsonb types. When an oversized ID that exceeded bigint range was introduced, the load stopped, and because it ran atomically, none of the 50,000 updates were applied. The tool also checks integer ranges before writing each batch so errors are reported clearly.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI agents can use your browser session instead of your password, experts say

APX, a free open-source desktop app, lets users run their own AI agents locally, including for tasks on websites. The article argues that the key question is who signs in, which browser session is used, and what the agent may do afterward, rather than whether the agent can open a browser. Users are advised to start with narrow tasks, verify the site address before entering credentials, and complete one-time codes or CAPTCHAs themselves. APX offers a browser handoff feature so a person can take over the tab for sensitive steps, though this requires a desktop browser rather than a headless one. The piece also cautions that a successful login does not confirm the task was completed, and recommends signing out and reviewing permissions once access is no longer needed.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How BreathTime's petition system works: proposals, admin review and signing

BreathTime is a small civic platform where registered users can propose petitions, sign them and leave supporting comments. Any logged-in user submits a proposal with a title, description and addressee, which is saved as pending and reviewed by a human administrator who can approve or reject it, with rejection comments sent back to the author. Only administrators publish approved petitions, setting the goal (1,000 signatures by default), and each petition is signed once per user, enforced both in code and in the database. Signatures can include an optional public comment, and the platform openly lists current limitations such as no real-time updates, no pagination in the admin list, no audit log and no automated moderation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Queue consumers need age-based alerts and idempotent sends to handle rate limits

A weekly digest pipeline can rely on a local token bucket placed before an idempotent send, backed by durable retries and a metric tracking the oldest queued work. Relying on queue depth alone is insufficient, since a consumer can appear healthy while missing the delivery window. The recommended approach is to record enqueue timestamps and alert when the oldest ready message exceeds the customer-facing delivery promise. Rate limiting decides when a request may start, while idempotency keys and durable state transitions prevent duplicate customer emails after timeouts. Teams should document the delivery window, normal drain rate, and maximum retry horizon in a runbook to set sensible thresholds.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Delivering Large CSV and PDF Exports to Browsers via Object Storage Presigned URLs

The article recommends generating large CSV or PDF exports outside the web request, storing the finished file as an immutable object, and letting browsers download it through short-lived presigned URLs. It argues that the application should manage authorization and export state, while object storage handles the byte transfer and the browser manages resumption. Separating these concerns prevents slow downloads from tying up application workers, rather than simply raising request timeouts. The author stresses that export readiness, URL expiry, and retry behavior must be tracked as distinct states, with retries using stable export identifiers and byte ranges to avoid duplicated work or mixing bytes from different object versions. The piece also notes that the exact validation method depends on the specific storage service and should be tested.

0 comments Read more at DEV Community