SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Querying a Partitioned Azure Blob Data Lake from PostgreSQL with pg_duckdb

0
·2 views

The article explains how the pg_duckdb extension lets PostgreSQL query a partitioned Parquet data lake stored in Azure Blob Storage as a single table. Its read_parquet() function accepts glob patterns and explicit file lists, so a query reads only the files it needs. Hive-style folder names such as year=2024 are exposed as real columns when hive_partitioning is enabled, and files can be pruned by partition. Examples use PostgreSQL 18.6 with pg_duckdb 1.1.0, showing that partition values are inferred as bigint and that union_by_name can align columns across files with differing schemas. The author notes that ARRAY[...] syntax is required for lists in PostgreSQL, since DuckDB's bracket syntax is rejected by the PostgreSQL parser.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Meta-learning method aims to keep aquaculture sensors accurate on low-power solar buoys

The article describes a technique called meta-optimized continual adaptation for water-quality monitoring in fish farms running on low-power autonomous devices. It grew out of a field trip to a tilapia farm, where solar sensor buoys suffered from membrane biofilm drift, algae-driven shifts in turbidity readings, and rapid battery drain. The author argues that aquaculture data is highly non-stationary, labels such as calibrated oxygen readings are scarce, and power budgets are tight, all of which constrain system design. The proposed approach combines meta-learning, continual learning, and edge-efficient inference so that a model can adapt quickly from few examples while retaining critical knowledge, such as detecting genuine low-oxygen events. The piece also notes that the author is still working through unresolved challenges.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Remote developer weekly release rhythm: Monday planning, midweek previews, Friday delivery

The article describes a fixed weekly workflow for working with a remote developer, aimed at making progress visible between calls. Each week begins with a short call to agree on what should go live by Friday, with large tasks split into independently workable pieces. Midweek, the client receives a preview link to test in-progress work and give feedback while changes are still inexpensive. Releases ship on Friday with plain-language notes covering what changed, what to check, and what comes next. The code stays in the client's own repository from the start, and the approach is said to suit ongoing product work rather than one-off fixes or projects lacking a clear goal.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Awas drops AI-proposed scam evidence unless quotes match pages verbatim

A developer built Awas, a tool that checks job ads for recruitment scam signs, for the Nebius Global AI Hackathon. The original plan was to find published reports about each advertised company, but only 8 of 20 known scam cases met the 12-case target. Reports also rarely named the recruiting company, so the author rescoped the verdict to rules quoted from official pages. Web search now only supplies context for families, not inputs to the rules. A code gate keeps model-proposed evidence only when its quote appears word for word in the fetched page, and the author published the failed coverage result.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Customer-Support Knowledge Base Crawl Limits: Lower Index Cost, Traceable Retrieval

A DEV Community article argues that a fintech support knowledge base should use staged retrieval with explicit collections, bounded queries, and traceable sources. It recommends indexing only approved help content and attaching tenant, access, and version metadata to every document. The author suggests treating crawl boundaries as a product rule, defining owners, URL prefixes, and deletion behavior for each source. Queries should name their collection, apply metadata filters before ranking, and enforce a top-k ceiling, since large values raise indexing, reranking, and context costs. The piece includes a Go example that lists vector collections through an API, and notes that any top-k limit should be tested against representative support questions.

0 comments Read more at DEV Community