Object Storage Explained: How S3 and Cloud Data Architecture Actually Works

·1 views

Object storage is a data architecture that stores information as self-contained units — each combining raw data, metadata, and a unique identifier — rather than using folder hierarchies or disk blocks. Popularized by Amazon S3 since its 2006 launch, the model supports objects up to 5 TB and is designed to scale to billions of items without performance loss. Unlike block or file storage, object storage uses a flat namespace called a bucket, making retrieval as simple as a single API call using a unique key. Because metadata travels with each object, there is no directory tree to maintain, which simplifies scaling across commodity hardware. The architecture achieves high durability — AWS S3 claims 99.999999999% annual durability — by distributing data across multiple zones using replication and erasure coding.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Read-Through and Write-Through Caching: How Redis Handles Data Sync Automatically

Read-through and write-through are two Redis caching patterns that shift data-loading and write logic from the application into the cache layer itself. In read-through caching, the cache automatically fetches missing data from the database on a miss, centralising the loading logic rather than repeating it across the application. Write-through caching ensures that every write updates both the cache and the database synchronously, keeping them consistently in lockstep at the cost of higher write latency. Together, the two patterns let applications treat the cache as their primary data interface, eliminating scattered invalidation logic. However, the approach is best suited to read-heavy workloads where consistency is critical, and may be overkill for simpler use cases where cache-aside suffices.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Step-by-Step Guide: How to Create and Connect to a Linux VM on Microsoft Azure

Microsoft Azure allows users to create Linux virtual machines through its cloud Infrastructure as a Service (IaaS) platform, eliminating the need for physical hardware. Users can set up a VM via the Azure Portal by configuring a resource group, selecting Ubuntu as the operating system, and enabling SSH and HTTP inbound ports. After deployment, it is recommended to extend the public IP idle timeout from the default 4 minutes to 30 minutes to avoid connectivity disruptions. The VM can then be accessed remotely from a local terminal using an SSH command with the assigned public IP address. This setup is particularly suited for beginners using a free-trial Azure account for learning or testing purposes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Redis Sets Explained: Fast Membership Checks and Built-In Set Operations

Redis sets are unordered collections of unique strings that enable instant membership checks and server-side set operations such as intersection, union, and difference. Commands like SISMEMBER run in constant time, making sets ideal for use cases including permission checks, deduplication, follower relationships, and unique visitor tracking. Unlike lists, sets automatically ignore duplicate entries, eliminating the need for application-side deduplication logic. Operations like SINTER and SUNION allow Redis to compute relationships between sets atomically, replacing loops in application code with single fast commands. For cases requiring only approximate unique counts at scale, Redis offers HyperLogLog as a memory-efficient alternative to full sets.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Redis Lists Explained: How to Build Simple Queues and Capped Feeds

Redis lists are ordered string sequences that support push and pop operations from either end, making them well-suited for basic job queues and recent-item feeds. Producers can add jobs using LPUSH while workers retrieve them via RPOP or the blocking BRPOP, which eliminates inefficient polling by making workers wait until a job is available. A key limitation is that plain list-based queues offer at-most-once delivery, meaning a job is lost if a worker crashes after popping it but before completing the task. The LMOVE command partially addresses this by atomically moving a job to a separate processing list, enabling crash recovery through manual bookkeeping. For use cases requiring strict delivery guarantees and acknowledgements, Redis Streams are the recommended alternative, while lists remain a practical choice for best-effort queues and bounded recent-activity feeds.

0 comments Read more at DEV Community