Apache Iceberg v3 Variant Type Uses Shredding to Speed Up JSON Analytics

·1 views

Apache Iceberg version 3 introduces a new Variant data type designed to eliminate the performance bottlenecks long associated with storing JSON in analytical tables. Previously, engineers had to choose between storing JSON as raw strings — which forced engines to parse entire documents on every query — or flattening JSON into rigid typed columns that required constant schema migrations. The Variant type addresses both drawbacks by combining flexible JSON storage with a technique called shredding, which extracts frequently queried fields into typed sub-columns while retaining the full document. This approach allows query engines to read only the relevant fields without parsing the entire JSON string, bringing performance closer to that of traditional typed columns. The feature targets a widespread pain point in data engineering, where semi-structured data from mobile apps, IoT sensors, and third-party APIs often resides in slow, costly JSON string columns.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Apache Iceberg V3 Deletion Vectors Explained: Faster, Cheaper Row-Level Deletes

Deleting individual rows in a data lakehouse is technically complex because object stores like Amazon S3 treat files as immutable — they can be written and read but never edited in place. Apache Iceberg V2 addressed this with 'delete files,' a merge-on-read mechanism that logged which rows to ignore rather than rewriting entire data files. While this made row-level changes practical, it shifted processing costs to readers and degraded performance under heavy workloads. Iceberg V3 introduced 'deletion vectors,' a more efficient approach that improves read performance and scales better under pressure. The shift from delete files to deletion vectors represents a significant architectural refinement that expands the range of workloads a lakehouse can reliably support.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How GCC Handles Function Returns and Stack Memory Allocation on x86-64

A technical deep-dive explains how GCC manages stack memory and function return values during compilation on x86-64 architecture. The article covers three return mechanisms defined by the System V AMD64 ABI: returning scalar values via a single register, returning small structs across multiple registers, and returning large structs via a hidden pointer supplied by the caller. Stack memory allocation is handled through standard x86 instructions, where GCC adjusts the stack pointer register to reserve space for a function's local variables. The piece also explains the role of the endbr64 instruction, a hardware-level security checkpoint from Intel and AMD's Control-flow Enforcement Technology that prevents Jump-Oriented Programming attacks. Sample C code and GCC disassembly output via objdump are used throughout to illustrate each concept.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer launches BrickSlider, a framework-agnostic carousel library with Tailwind support

A developer has released BrickSlider, an open-source slider library designed to work independently of any JavaScript framework or styling system. The library handles carousel behavior — including touch gestures, loop logic, and responsive layouts — while leaving visual design entirely to the consuming project. BrickSlider uses the Web Animations API for motion control rather than inline style mutations, aiming for cleaner animation management. It supports plain CSS, Tailwind, CSS modules, and CDN-based setups, requiring only the DOM to mount. The project is split into focused packages — a core engine, accessibility helpers, a story-style plugin, and Tailwind-oriented utilities — so developers can opt into features as needed.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

CLI tool 'Secably' catches expiring TLS certs and weak security headers in CI

A lightweight command-line tool called Secably lets developers automatically check TLS certificate expiry, HTTP security headers, DNS posture, and subdomain exposure directly from the terminal. The tool can be integrated into CI pipelines via a one-line GitHub Action that fails the build if a certificate is nearing expiry or if security headers fall below a set grade. Secably scores HTTP response headers from A to F across six key headers, including HSTS, Content-Security-Policy, and X-Frame-Options, and flags stack-leaking headers like verbose Server values. It also checks DNS records for SPF, DMARC, and DNSSEC misconfigurations, and discovers subdomains passively using public Certificate Transparency logs without sending traffic to the target. The tool runs entirely locally with no third-party data sharing, and supports JSON output for integration with other pipeline tools.

0 comments Read more at DEV Community