Engineer cuts AWS Glue pipeline runtime by 56% with four targeted optimizations

·2 views

A software engineer reduced the end-to-end runtime of a production AWS Glue and PySpark pipeline — handling Oracle-to-S3 ingestion, encryption, and downstream transfer — by approximately 56% without altering any business logic. Key gains came from raising Step Functions Map concurrency from 30 to 50, eliminating redundant multi-pass file rewrites in the publish layer, and threading CPU-bound ZIP compression while lowering DEFLATE to level 1. Two additional investigations yielded lessons rather than wins: parallelizing GPG encryption caused out-of-memory crashes due to multi-GB files being loaded fully into RAM, and an SFTP transfer job proved bandwidth-bound rather than concurrency-limited. The engineer ultimately moved the SFTP job to a lighter Glue Python Shell instance, cutting infrastructure cost while maintaining the same throughput.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Send Telegram Bot Messages in PHP Using cURL with Full Error Handling

Developers can send Telegram bot messages using PHP's cURL extension by calling the sendMessage API method over HTTPS. A minimal helper function handles the full request lifecycle, including reading the bot token from an environment variable, building the payload, and setting connection timeouts. The function performs layered error checks: verifying the HTTP status code, safely decoding the JSON response, and confirming that Telegram's 'ok' field returns true. If any of these checks fail, the function throws a descriptive runtime exception to help diagnose the issue. The guide focuses strictly on the sendMessage call and does not cover webhook management, SDK functionality, or state persistence.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Binance Order Flow Data Boosts Polymarket BTC Trade Accuracy to 70%, Study Finds

A study analyzing 2,833 five-minute Polymarket markets for BTC, ETH, and SOL found that combining the late-stage market leader with Binance aggressive order flow data significantly improved historical prediction accuracy. When price direction and order flow aligned, the market leader won approximately 70.2% of the time, compared to just 57.6% when using price signals alone. Conversely, when price and order flow diverged, the leader won only 41.3% of the time, suggesting the underlying buying or selling pressure told a different story than the price chart. The research proposes an automated trading bot architecture that separates data collection, signal generation, and order execution into distinct components. Researchers cautioned that these figures reflect historical results only and do not guarantee future trading performance.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Free Browser-Based Fence Calculator Using Vanilla JavaScript

A developer has created a lightweight, browser-based fence calculator called the ItaliaLegacy Fence Calculator, built entirely with HTML, CSS, and vanilla JavaScript. The tool estimates key fencing materials — including sections, panels, posts, and rails — based on user-entered measurements such as total fence length, gate widths, post spacing, and rails per section. All calculations run locally in the browser, meaning no data is sent to a server. The calculator accounts for gate openings by subtracting gate widths from the total fence length before computing material estimates. It is designed as a planning aid rather than a precise purchasing guide, as real-world variables like corners and direction changes can affect actual material needs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Kaggle Benchmark Tests AI Models on Booking Conflict Replay Logic

A new Kaggle benchmark called Reservation Replay challenges AI models to correctly handle booking conflicts, specifically testing whether a failed request remains failed even after conditions change. The benchmark uses fictional rooms and integer time intervals across 8 base traces and 4 dependent metamorphic variants, scored by exact trace success without an LLM judge. Google's Gemini 3.7 Flash achieved a perfect 12/12 score, while Gemini 2.5 Flash scored only 4/12, primarily due to output formatting failures rather than logical errors. A scoring policy update was introduced mid-evaluation after Gemini 2.5 Flash triggered a ResponseParsingError, highlighting how format compliance can significantly skew benchmark results. The author cautions that the small pilot size and dependent variants are insufficient to establish stable model rankings, but the reproducible fixtures and preserved run evidence make the findings verifiable.

0 comments Read more at DEV Community