SShortSingh.
Back to feed

Fixing BigQuery query structure cut a client's bill by 72% in three weeks

0
·1 views

A data engineering team reduced a client's Google BigQuery costs by 72%, saving around £2,000 per month, in just three weeks by restructuring how queries were written rather than reducing data volume. BigQuery's on-demand pricing charges per byte scanned, meaning inefficient query design — not data size or traffic — is the primary cost driver. The biggest savings came from eliminating SELECT * statements, partitioning and clustering tables by commonly filtered columns, and enforcing partition filters to prevent accidental full-table scans. Several scheduled queries were also found to be feeding tables that no longer had any active readers and were paused and eventually deleted without any user complaints. The team noted that bytes scanned directly affects both cost and query latency, meaning structural fixes simultaneously reduced bills and improved dashboard performance.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

GitHub Launches Human-in-the-Loop Controls for AI Agent Automation in Issues

GitHub announced agent automation controls for GitHub Issues on July 23, 2026, currently available in public preview. The feature allows AI agents to attach confidence levels — high, medium, or low — to proposed issue actions such as labeling, triaging, or closing. Repository administrators can set a minimum confidence threshold, below which agent suggestions require human approval before being applied. Each suggested action also includes a rationale, helping reviewers understand why the agent proposed a change and make faster, informed decisions. The controls aim to bridge the gap between slow manual triage and risky fully autonomous automation by keeping humans involved where judgment matters most.

0
ProgrammingDEV Community ·

Developer hits four dead ends trying to build Apple Shortcuts programmatically

A developer attempted to automate publishing to a REST API using Apple Shortcuts, building a three-step shortcut that lets Siri dictate and POST content directly to an endpoint. While the shortcut worked fine when built manually on-device, every attempt to create, sign, or deploy it through code failed. The macOS 'shortcuts sign' CLI rejected hand-written property lists and, on macOS 14.6, returned an iCloud authentication error even on valid files with iCloud fully enabled. The URL import scheme only accepted links hosted on Apple's own icloud.com domain, blocking any self-hosted or third-party file imports. The only partial workaround discovered was reverse-engineering an undocumented iCloud API endpoint that returns unsigned, readable shortcut plists from shared links.

0
ProgrammingDEV Community ·

A User's Journey Into the Hidden World of Mobile App Testing

A regular app user shares how their perspective on software updates shifted after exploring a mobile testing platform. Previously indifferent to the effort behind app development, they assumed testing simply meant checking that an app opened and ran without crashing. Digging deeper revealed a complex process involving functional testing, exploratory testing, and cross-platform compatibility checks across different devices and operating systems. Testers must account for everyday edge cases such as denied permissions, mid-payment disconnections, and varying screen sizes before an app reaches users. The experience highlighted that effective testing is largely invisible — when apps work seamlessly, the extensive preparation behind them goes unnoticed.

0
ProgrammingDEV Community ·

Five Common Docker Security Mistakes and How to Avoid Them

A developer with several years of Docker experience has outlined five recurring security mistakes observed in containerized application deployments. Key errors include assuming small self-hosted projects remain undiscovered, when in reality automated bots scan public IP space within minutes of a server going live. Many developers also mistakenly treat HTTPS as comprehensive security, when encryption only protects data in transit and does not block malicious requests like SQL injection or credential stuffing. Neglecting Layer 7 protections, such as Web Application Firewalls, leaves application endpoints exposed to HTTP-level attacks that infrastructure-level hardening cannot address. The article urges developers to close unnecessary ports, validate application inputs, and layer security measures rather than relying on any single control.