SShortSingh.
Back to feed

Python Futures Explained: How concurrent.futures Speeds Up IO-Bound Apps

0
·1 views

Python's concurrent.futures module allows developers to run IO-bound and CPU-bound tasks asynchronously using Future objects as placeholders for pending results. A Future tracks a task's lifecycle through states such as PENDING, RUNNING, and FINISHED, and stores the final return value or any exception once execution completes. Unlike synchronous code where tasks run sequentially, offloading work to background threads via ThreadPoolExecutor lets multiple operations run in parallel, cutting total wait time to that of the longest single task. The module suits use cases like scraping web pages, querying multiple databases, and processing images across CPU cores. A practical e-commerce example illustrates how product-fetching tasks with simulated network latency can be submitted concurrently and handled through callbacks and completion utilities like as_completed and wait.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Researchers Detail EEPROM Hijacking Method Targeting 8051 Microcontrollers

A technical methodology has been outlined for intercepting and redirecting execution flow in 8051-architecture microcontrollers, which remain widely used in constrained embedded systems. The technique involves inserting a binary hook — a long jump instruction — at the start of an SDCC-compiled validation function to redirect execution to a custom assembly routine stored in unused memory, known as slack space. A key challenge addressed is byte-alignment: replacing a 4-byte prologue with a 3-byte LJMP instruction requires a padding NOP to prevent program counter corruption and system crashes. The injected assembly routine conditionally bypasses a critical logic or security check only when a specific trigger value is detected in memory, otherwise resuming normal program execution. The approach is noted for its minimal memory footprint, making it particularly relevant in environments where available storage is measured in just a few kilobytes.

0
ProgrammingDEV Community ·

Key Engineering Lessons from Building Reliable Payments Infrastructure

A payments infrastructure engineer has shared hard-won lessons from scaling financial systems, emphasizing that unlike typical software, payment failures have immediate and serious consequences for users. The most critical lesson is idempotency: systems must be designed to handle duplicate requests safely, since retries, double-taps, and repeated webhooks are routine occurrences rather than edge cases. The engineer warns that a naive check-then-write approach leaves race conditions open, and that only database-level unique constraints can reliably prevent duplicate charges. Additional lessons cover treating payment gateways as failure-prone vendors, proactive monitoring to detect outages before customers do, and the unglamorous but essential work of ledgers, reconciliation, and state machines. The piece concludes that each of these principles involves tradeoffs that often conflict with one another, requiring careful engineering judgment.

0
ProgrammingDEV Community ·

Why yfinance Fails to Fetch Natural Gas Futures Data and How to Fix It

Developers using the yfinance Python library have reported recurring failures when downloading natural gas futures data via the NG=F ticker on Yahoo Finance. The issue stems from yfinance being an unofficial scraping tool, meaning any backend change by Yahoo can silently break data retrieval without warning. Futures symbols like NG=F are especially prone to these disruptions, with multiple known issues logged by the community over the years. Three short-term fixes include upgrading the library, adding retry logic with backoff, and switching between yf.download() and yf.Ticker().history() methods. For production pipelines requiring reliable data, the author recommends migrating to a dedicated natural gas price API rather than depending on an unsupported scraping workaround.

0
ProgrammingDEV Community ·

DNS gateway hit 500 QPS wall due to missing Bloom filter and SQLite bottleneck

HydraDNS, an open-source DNS security gateway built in Go, was falsely documented as using a Bloom filter for blocklist lookups, when in reality every DNS query triggered a SQL COUNT against a 92,000-row SQLite table. The shared single database connection serialized both blocklist checks and query logging, capping throughput at around 500 queries per second regardless of available CPU headroom. The developer discovered the phantom Bloom filter never existed in code — a planning note had gradually become accepted fact across multiple documents. Replacing the SQL lookup with an in-memory atomic domain map boosted throughput nearly 19x, from ~500 to ~9,500 QPS. However, removing that first bottleneck exposed a second: per-query goroutines for logging overwhelmed the single database connection at high load, causing memory to spike toward 2GB and risking out-of-memory crashes on target hardware.

Python Futures Explained: How concurrent.futures Speeds Up IO-Bound Apps · ShortSingh