SShortSingh.
Back to feed

How Engineers Partitioned a Billion-Row Logistics Table in Production

0
·1 views

A backend engineering team at a nationwide logistics platform faced severe database performance issues after their core tracking events table surpassed one billion rows, rendering traditional indexing ineffective. The table logged every parcel scan and status change, growing continuously as tens of thousands of orders were processed daily. The team implemented range-based table partitioning in PostgreSQL, splitting data into monthly partitions keyed on a timestamp column to enable partition pruning — allowing the database to skip irrelevant date ranges entirely. A critical constraint they encountered was that the partition key must be included in the primary key, which broke ORM assumptions and complicated foreign key relationships. The engineers emphasized that the real trigger for partitioning should be access pattern alignment, not table size alone — nearly all their queries were already scoped to specific date ranges, making the approach a natural fit.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How to Scale Real-Time Delivery Tracking for 10,000 Reconnecting Users

Engineers building real-time delivery tracking maps face a core challenge: maintaining accurate state when users lose and regain connectivity across different networks. The proposed architecture separates three distinct contracts — durable delivery state, transient room presence, and the connection layer — ensuring that truth lives in the event log, not the connection. Each delivery stream uses a monotonically increasing sequence number, allowing clients to store their last applied position and request only missed events upon reconnection, falling back to a full snapshot when gaps cannot be replayed. Versioned event envelopes enable rolling releases by letting old and new consumers coexist safely, while incompatible schema changes require explicit new event types rather than silent field reuse. The result is a system where dropped connections delay updates but cannot corrupt state, and duplicate events waste bandwidth but cannot reverse progress.

0
ProgrammingDEV Community ·

The start of a new journey

Have you ever wondered why we keep learning advanced things that probably might not be applied properly where we came from? As someone who came from a developing country where resources are not being served on a gold plate. In fact, even if you have all the necessary knowledge to make a change but one thing comes up with no answer, how can we implement our knowledge gained abroad with no funding and no equipment to help us contribute to the blooming of our beloved country? I guess our parents worked hard to actually send us abroad, not to return to our country but instead to find a way to make

0
ProgrammingDEV Community ·

OpenAI Usage API Now Supports API Key Grouping for Token and Cost Reconciliation

OpenAI's August 4, 2026 API changelog introduced API-key filtering and grouping across its usage and cost endpoints, enabling developers to attribute token activity and spending to specific keys. Because the two endpoints return different datasets, a simple inner join risks hiding unmatched or unattributed rows, making a full-outer join on date and API-key ID the safer approach. Both endpoints support daily bucketing and pagination, and consistent grouping dimensions across both sources are required to ensure row-level compatibility. A .NET implementation demonstrates the pattern using synthetic data, flagging each row as matched, usage-only, or cost-only to keep gaps visible rather than concealed. Notably, the approach deliberately avoids estimating costs from token counts, since cost buckets may include line items beyond completion tokens.

0
ProgrammingDEV Community ·

FastAPI File Uploads: The Gateway to Document-Based AI Applications

FastAPI simplifies file uploads for AI applications using the python-multipart library and two key classes: File and UploadFile. The UploadFile class provides useful attributes such as filename, content type, and async file reading, making it well-suited for processing documents. Developers can save uploaded files to disk using Python's binary write mode, which supports non-text formats like PDFs and images. This upload workflow forms the foundation of AI systems such as RAG pipelines, resume analyzers, and medical report processors. The tutorial is Part 8 of a FastAPI series aimed at AI engineers building document-centric applications.

How Engineers Partitioned a Billion-Row Logistics Table in Production · ShortSingh