How Erasure Coding Replaced Triple Replication in Modern Object Storage

·1 views

Traditional object storage relied on three-way replication for durability, but at cloud scale storing 30 PB to protect 10 PB of data became a massive financial burden. Engineers sought a way to ensure data protection without keeping full duplicate copies, leading to the adoption of erasure coding. This technique splits an object into data fragments and adds mathematically derived recovery fragments, allowing lost pieces to be reconstructed without storing complete duplicates. An 8+4 erasure coding scheme, for example, requires only 150 TB of storage to protect 100 TB of data, compared to 300 TB under three-way replication. The trade-off is increased computational complexity during recovery, as rebuilding missing fragments demands CPU resources and network coordination rather than a simple file copy.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Pilum CLI Goes Production-Ready After 3 Months and 40+ Fixes

Developer open-sourced Pilum in December 2025, a multi-cloud deployment CLI that targets Cloud Run, Lambda, Azure, Cloudflare Pages, npm, Homebrew, and Docker Hub from a single configuration file. Over the following three months, more than 40 pull requests were merged as the tool moved from a working prototype to powering all production deployments at SID Technologies. A critical wave-based deployment bug caused workers to start subsequent deployment waves before earlier ones had fully completed, leading to intermittent npm publish failures due to race conditions. A separate bug silently swallowed shell command errors, allowing failed deployments to be reported as successful without propagating errors to the orchestrator. Both issues were resolved through an orchestrator rewrite and stricter wave barrier enforcement, with the developer noting that major bugs consistently surfaced about a month after each feature sprint.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Replicate GitLab's Centralized CI/CD Pipelines in GitHub Actions

Developers migrating from GitLab to GitHub Actions face a structural challenge: GitLab allows a central 'pipelines' repository to be referenced across projects via the 'include' keyword, while GitHub requires workflows to be defined within each repository individually. GitHub's 'uses' keyword offers partial reusability but operates in its own scope, requiring explicit inputs and outputs, unlike GitLab's seamless local-context inclusion. To bridge this gap, developers can use composite actions to bundle reusable steps or leverage GitHub Apps for organization-wide CI/CD enforcement. Smaller teams are advised to adopt reusable workflows with tagged versions, while larger organizations with compliance needs may benefit from GitHub Apps despite the added administrative overhead. Without careful versioning and scoping management, centralized pipelines in GitHub risk inconsistent behavior and increased maintenance burden across dependent projects.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Doctor Directories Fail Users and How to Navigate Them Smartly

Online doctor directories frequently display inaccurate information — including outdated phone numbers, departed physicians, and false availability — due to underlying data engineering failures rather than negligence. These platforms pull records from multiple sources such as insurance networks, licensing boards, and hospital affiliations, each updating on different schedules and using inconsistent formats, creating entity-resolution errors. A quarterly sync cycle means a significant share of listings are wrong at any given time, and an ongoing physician shortage makes phantom listings especially costly for patients. Experts recommend treating directory results only as a starting point, verifying credentials directly through state medical board websites, and calling practices to confirm insurance participation and patient availability. This multi-step approach transforms an unreliable aggregator into a useful discovery tool, though it demands considerably more effort than a simple search.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Graduate Chooses Action Over Readiness to Build Skills and Career

An AI and Data Science graduate has decided to stop waiting until they feel fully prepared and instead begin building projects and applying for jobs in public. The writer had long believed a stronger portfolio was a prerequisite for starting, but came to recognize that confidence develops through action rather than before it. They are currently deepening their AI knowledge beyond coursework, developing practical projects, and documenting their learning process openly. Their blog is framed as a commitment to consistency over perfection, with plans to share projects, mistakes, and career experiments. The initiative is aimed at connecting with others who are similarly navigating early-career uncertainty in tech.

0 comments Read more at DEV Community