SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

BigQuery, Spark, and Trino Can Now Query the Same Iceberg Table on GCP

0
·1 views

A technical demonstration on Google Cloud Platform shows how BigQuery, Apache Spark, and Trino can simultaneously read and write a single Apache Iceberg table without duplicating data. The setup uses Google Cloud's Lakehouse for Apache Iceberg service — formerly known as BigLake — which provides a native REST Catalog to coordinate table versions across query engines. Apache Iceberg acts as an open metadata layer over Parquet files stored in Google Cloud Storage, enabling ACID transactions, time travel, and schema evolution. The approach directly addresses a common enterprise problem where separate teams store redundant copies of the same data in different tools, leading to higher storage costs, inconsistent records, and fragmented access controls. The experiment was validated on real GCP infrastructure, including a captured HTTP 409 conflict error demonstrating Iceberg's optimistic concurrency control in action.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How One SaaS Team Replaced Ad-Hoc AI Prompting With a Structured Operating System

A SaaS operator has published a detailed guide outlining how their team moved away from one-off AI prompting toward a structured, repeatable operating system for running multiple software products. The system is built around two core layers: a set of organised playbooks covering key founder workflows such as validation, building, launching, and growth, and an engineering stack that automates and logs those playbooks on a schedule. The motivation came after inconsistent AI outputs — including hallucinated competitor features and off-target content — revealed that the problem was structural rather than a matter of prompting skill. Research cited in the guide suggests that teams relying on unstructured prompts face higher error and hallucination rates compared to those using grounded, repeatable processes. The guide also covers how the team integrates a CRM with AI workflows and built an automated Reddit trend research tool to support content operations.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Why 'When Did This Change Happen?' Has No Single Answer in Modern Databases

Determining exactly when a database change occurred is a complex, multi-layered question critical to replication, auditing, and application-log correlation. In MVCC-based databases, a single row update involves at least six distinct moments — from transaction start and read point selection to commit visibility and client acknowledgment — none of which are interchangeable. PostgreSQL tracks these using transaction IDs, snapshot coordinates, and Log Sequence Numbers, while Oracle consolidates many functions into its System Change Number, though it still relies on additional identifiers. Distributed systems like YugabyteDB use hybrid logical clocks, while SQL Server, MySQL, and MongoDB each split these responsibilities differently. The core insight is that no single timestamp or coordinator answers all ordering questions, and the right one to use depends entirely on what the application is actually asking.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

EU Moves to Make Huawei Ban Legally Binding Across Critical Infrastructure

The European Commission reissued its recommendation in May 2026 to exclude Huawei and ZTE from critical telecom infrastructure, reinforcing a stance it first took in 2020. A January 2026 revision to the EU Cybersecurity Act would transform six years of voluntary guidance into a binding legal obligation, proposing a 36-month phase-out window covering mobile, fixed broadband, and transport infrastructure. Only 13 of 27 EU member states had taken concrete action under the original 2020 framework, highlighting weak compliance with voluntary measures. The proposed legislation is still under review by the European Parliament and Council, with key questions remaining around the phase-out timeline and how heavily exposed nations like Germany may seek to negotiate its terms. Cost estimates for the transition vary sharply — from €10–13 billion by the Commission to €35 billion by industry groups — a gap analysts say reflects the absence of a shared model for mapping the full scope of vendor dependencies involved.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer builds MCP server to track brand visibility across AI answer engines

A developer has launched Cituna, a tool designed to monitor how often a product or brand is cited by AI-powered answer engines such as ChatGPT, Perplexity, Gemini, Claude, Grok, and Google's AI search features. The tool runs daily prompt simulations across all seven platforms and records which brands each engine mentions in response to buyer-style queries. It exposes this data via the Model Context Protocol, allowing users to query their AI visibility directly inside Claude without switching tools. Cituna also integrates with Google Search Console to highlight cases where a product ranks well on traditional search but remains absent from AI-generated answers. A free first scan is available, and the tool offers both an MCP interface and a REST API for developers who prefer to build on top of the data.

0 comments Read more at DEV Community
BigQuery, Spark, and Trino Can Now Query the Same Iceberg Table on GCP · ShortSingh