Key storage requirements determined by four lines in service manifest

·1 views

A software engineer describes a method where storage decisions for services are based on four specific requirements defined before coding begins. These requirements include daily and total row counts, read-write ratios, row lifetime, and tenant isolation considerations. The approach is designed to force explicit, early decisions about storage architecture rather than leaving them to chance during development. This is especially critical in systems where data is immutable and only accumulates via versioning, never being deleted or overwritten. The four lines directly inform technical choices like partitioning, sharding, and database-per-tenant models.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Software developers reflect on the enduring power of simple, single-purpose functions.

An article on DEV Community explores the aesthetic and practical value of simple functions in programming. It argues that a well-designed function performs a single, clear task without unnecessary complexity. Such functions create reliable boundaries and abstractions within larger software systems. Their simplicity allows developers to understand, trust, and reuse them without remembering internal details. This focus on clarity is presented as a foundational principle of effective software construction.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Data pipeline loaded erroneous $4.5 million order without raising errors

A developer conducted an experiment by loading a sabotaged data batch with 12 planted defects into a test data warehouse. The load succeeded without errors, and one erroneous order valued at $4.5 million was included, causing reported revenue to spike from about $395,751 to over $4.9 million. Standard and custom data quality tests in the dbt pipeline correctly identified all 12 defects, including the outlier amount, after the load. However, the pipeline's core loading process, focused only on moving valid data formats, did not inherently block the erroneous row from being ingested. The experiment highlights a distinction between data movement and quality validation, where checks run after loading.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Expert advises against fine-tuning AI models as initial solution

An AI expert argues that fine-tuning models is often an expensive and ineffective first step to address performance issues. The core problem is usually a lack of proper context due to flawed retrieval systems, contradictory instructions, or poorly formatted inputs, which fine-tuning does not fix. The article recommends first improving prompts and retrieval, and establishing a robust evaluation dataset to measure any performance gap. Fine-tuning should only be considered for narrow, stable, high-volume tasks where evaluations confirm its necessity, after other methods have been exhausted.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Guide advises using fictional data for hackathon demos, not real identities

A developer guide recommends building hackathon demo data with entirely invented records. This approach protects privacy by avoiding the use of real emails, photos, or account histories in repositories or presentations. The guide suggests starting with a minimal dataset that demonstrates core application relationships, like tasks belonging to teams. It advises using reserved domains like example.com for email addresses and storing this synthetic data separately from live services. This practice ensures demos are convincing without risking real user data.

0 comments Read more at DEV Community