Engineer cuts AWS Glue pipeline runtime by 56% with four targeted optimizations

·22 views

A software engineer reduced the end-to-end runtime of a production AWS Glue and PySpark pipeline — handling Oracle-to-S3 ingestion, encryption, and downstream transfer — by approximately 56% without altering any business logic. Key gains came from raising Step Functions Map concurrency from 30 to 50, eliminating redundant multi-pass file rewrites in the publish layer, and threading CPU-bound ZIP compression while lowering DEFLATE to level 1. Two additional investigations yielded lessons rather than wins: parallelizing GPG encryption caused out-of-memory crashes due to multi-GB files being loaded fully into RAM, and an SFTP transfer job proved bandwidth-bound rather than concurrency-limited. The engineer ultimately moved the SFTP job to a lighter Glue Python Shell instance, cutting infrastructure cost while maintaining the same throughput.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

F/OSS Comics #13 Traces the Origins of BSD Unix in Part Three

The thirteenth installment of the F/OSS Comics series continues its illustrated history of open-source software with its third part on BSD Unix. This chapter focuses on the beginnings of BSD Unix, the Berkeley Software Distribution derived from AT&T's Unix. The comic is presented in a narrative, comic-strip format aimed at making the software's history accessible to general readers. The post was shared on Hacker News, where it drew a small amount of attention from the community.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

UIDAI Unveils Lighter Sub-AUA/Sub-KUA Route for Low-Volume Aadhaar Users

On 29 September 2026, UIDAI announced five initiatives at Aadhaar Samvaad in Kolkata, including a new LITE framework for Sub-AUAs and Sub-KUAs. Sub-AUAs and Sub-KUAs access Aadhaar services through an existing AUA or KUA, which handles Yes/No authentication and e-KYC respectively. The LITE framework is intended to simplify and speed up onboarding for entities with lower authentication volumes, including government and eligible non-government bodies. Under existing rules, AUAs and KUAs need UIDAI approval before appointing a Sub-AUA or Sub-KUA, and they remain jointly liable for compliance. The article notes that UIDAI has not yet released full details of the LITE route, so regulated entities such as lenders and fintechs are advised to prepare while awaiting clarification.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Money Tracker app explains bill import without bank logins, using confirm-before-save

The developer of Money Tracker, an expense app for Android and iOS, describes its statement and bill import feature, which is available on the Premium tier. Users export a file such as a PDF or CSV from their bank or card issuer, then upload it to the app, which parses it into draft entries. Each draft must be reviewed and confirmed by the user before it is saved to their records. The developer stresses that the app never asks for bank credentials and does not maintain ongoing connections to accounts, unlike bank-sync services. The post presents this as a narrower, more controlled alternative to automatic open-banking sync.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Contact form shows success but enquiries go missing: how to trace one test message

A contact form can display a success message while the enquiry never reaches the team's mailbox. The recommended approach is to follow a single identifiable test submission through the form, the sending route, and the receiving mailbox, rather than changing the mail plugin or DNS first. A successful wp_mail() return only means the sending method processed the request without error, not that the recipient got the email. Form plugins may also have demo or skip-mail settings that show success without sending, so these should be checked. Each finding, such as no matching record, a sending error, or provider acceptance, points to a different next check.

0 comments Read more at DEV Community