SShortSingh.
Back to feed

How Facebook Serves Personalised News Feeds to 3 Billion Users in 300ms

0
·1 views

Facebook's News Feed must load a unique, personalised content stream for over three billion users in under 300 milliseconds, a challenge that required building one of the most complex distributed systems in existence. To manage social connections at scale, Facebook developed TAO, a geographically distributed graph database that stores users, posts, and relationships as nodes and edges, enabling social graph traversal in microseconds without costly database joins. Delivering new posts to friends involves a tradeoff between Fan-out on Write, which pre-populates friends' caches instantly, and Fan-out on Read, which assembles the feed on demand. The push model works well for ordinary users but breaks down for celebrities with hundreds of millions of followers, where a single post could trigger massive simultaneous writes — a problem known as the Thundering Herd. Facebook resolved this with a hybrid fan-out architecture that applies different delivery strategies depending on a user's follower count.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Seven Lessons From Migrating a Multi-Tenant SaaS Dashboard From MUI to Shadcn

An engineering team replaced Material UI with Radix and Shadcn across a multi-tenant SaaS dashboard over the past quarter, reducing bundle size and simplifying their design system. The migration involved six teams and dozens of production components, and proved far messier than typical online tutorials suggest. Key issues included focus management differences in dialogs, incorrect component mappings such as using Drawer instead of Sheet, and Radix portal rendering bugs caused by CSS transform properties on ancestor elements. The team also had to build a custom multi-select and async-select solution on top of Radix Popover and Command, since Radix Select lacks native async and multi-value support. Their main takeaway was that a real-world migration rarely achieves full coverage, and maintaining clear escape hatches for complex legacy components is more practical than forcing a complete rewrite.

0
ProgrammingDEV Community ·

How PostgreSQL Internally Processes a SELECT Query: From SQL to Rows

When a PostgreSQL query like SELECT * FROM users is submitted, the database runs it through several internal stages before returning results. First, a parser checks syntax and builds a parse tree, then an analyzer verifies that referenced tables and columns actually exist. A rewriter may transform the query, after which the planner evaluates possible execution strategies — such as a full table scan versus an index scan — and picks the least costly one. The executor then carries out the chosen plan, retrieving data via a buffer cache called shared_buffers before falling back to disk if needed. Understanding this pipeline helps developers use tools like EXPLAIN more effectively to diagnose and improve query performance.

0
ProgrammingDEV Community ·

Developer finds 96 routing policy violations in AI agent logs, all off the main thread

A software developer auditing their AI agent's token usage discovered that 96 out of 425 routing decisions — about 22.6% — deviated from their stated policy, costing an extra $1,248.13. All deviations occurred in subagent threads, not on the main thread, revealing a gap between intended and actual model selection behavior. The developer maintains a 30-line YAML routing policy and an append-only trace log, comparing expected versus actual model tiers per decision unit. An early policy draft was found to be misaligned with actual intent, prompting a correction before measurements were taken to avoid scoring behavior against a flawed baseline. Separately, trace logs revealed an 18-day gap in Claude Fable 5 usage that appeared to reflect natural session drainage rather than a deliberate choice to stop using the model.

0
ProgrammingDEV Community ·

Ethical Web Scraping: Why Respecting Robots.txt and Rate Limits Is Non-Negotiable

Web scraping is legally permissible for publicly accessible data under a 2023 US court ruling in hiQ Labs v. LinkedIn, but only when site terms of service and authentication boundaries are respected. A 2024 Zyte survey found that 68% of scraping projects fail within the first month due to IP blocks, with a further 23% failing from legal threats. Experts warn that aggressive crawling behaviors — such as sending dozens of requests per second or spoofing browser identities — trigger firewalls, IP bans, and potential violations of the US Computer Fraud and Abuse Act or EU GDPR. Best practices include parsing robots.txt files before crawling, maintaining request delays of one to five seconds, and using transparent bot identifiers in User-Agent strings. A 2022 Cloudflare study found that crawlers observing a delay of at least one second encounter 90% fewer bot-blocking events than those making sub-second requests.

How Facebook Serves Personalised News Feeds to 3 Billion Users in 300ms · ShortSingh