SShortSingh.
Back to feed

Apache Iceberg v3 Adds Native Geometry Types to Fix Decade-Old Spatial Data Gap

0
·1 views

Apache Iceberg format version 3 introduces two native primitive types — geometry and geography — giving spatial data the same first-class treatment as timestamps and integers within lakehouse tables. For nearly a decade, teams stored coordinates either as two separate double columns or as opaque binary blobs, meaning query engines had to scan entire datasets without any spatial file-skipping. The new types pair with Apache Parquet 2.11's matching logical types to embed coordinate reference system metadata directly in the schema, eliminating silent data corruption caused by undocumented CRS assumptions. Per-file bounding-box statistics are now stored in Iceberg manifests, allowing engines to skip irrelevant files before reading any geometry data. The changes are defined at the specification level and apply across any compatible engine, not just Dremio, which currently ships Iceberg v3 support.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How Apache Iceberg's Puffin Format Solves Query Optimization Gaps in Large Tables

Apache Iceberg's Puffin is a lightweight container file format designed to store statistics that do not fit inside Iceberg's standard manifest files. Manifests efficiently hold per-file scalar metrics like row counts and value bounds, but cannot accommodate larger structures such as distinct-value sketches or row-level delete bitmaps. Without distinct-value estimates, query optimizers are forced to guess join strategies, potentially causing unnecessary shuffling of terabytes of data. Puffin addresses this by housing two blob types: Theta sketches for estimating distinct column values, and deletion vectors for tracking row-level deletes introduced in Iceberg format version 3. The format uses a simple structure — a magic number, opaque data blobs, and a JSON footer — allowing query engines to access costly statistics without bloating the manifests read on every query.

0
ProgrammingDEV Community ·

OpenAI Launches Prism, a Free AI-Powered LaTeX Workspace for Researchers

OpenAI has introduced Prism, a free cloud-based writing workspace built on its GPT-5.2 model, designed to help scientists draft, revise, and prepare research papers for publication. The platform combines a LaTeX editor with document-aware AI assistance, keeping equations, citations, figures, and references within a single working environment. Prism evolved from Crixet, a cloud-based LaTeX platform that OpenAI previously acquired and redeveloped. The workspace is currently available at no cost to personal ChatGPT account holders across Free, Go, Plus, and Pro tiers, with no stated seat limits for personal use. OpenAI has indicated that paid features and expanded access for Business, Enterprise, and Education users are planned, though no pricing details or firm rollout dates have been announced.

0
ProgrammingDEV Community ·

Why Measuring Pull Requests May Be Hurting Your Software Team's Performance

A software engineering advocate argues that measuring pull request metrics—such as duration and throughput—is counterproductive because PRs inherently slow down code integration and delivery. Citing a DX study of over 500 organizations, the author notes that the longer PRs remain open, the less code developers ship, particularly in high-performing teams. Rather than optimizing a practice that introduces delays, the author urges engineering leaders to adopt DORA metrics—deployment frequency, lead time for changes, change failure rate, and time to restore service—as more meaningful indicators of delivery performance. The piece contends that continuous integration with trunk-based development is a superior alternative, supported by over a decade of DORA research showing that speed and stability improve together. Using a marathon analogy, the author warns that measuring and rewarding PR efficiency is akin to timing tea service during a race—optimizing a stop that should not exist in the first place.

Apache Iceberg v3 Adds Native Geometry Types to Fix Decade-Old Spatial Data Gap · ShortSingh