Adding a fourth memcached server moved most cache keys and overloaded a product database
A PHP storefront team added a fourth memcached server to its three-server cache on a Monday morning before a peak sales season. Because the PHP Memcached extension used modulo hashing by default, most keys were reassigned to servers that did not hold them, so the cache hit rate dropped from about 97 percent to roughly 25 percent. The product database's read load rose more than twentyfold, and product pages timed out for around eighteen minutes until the cache refilled. The team later switched to consistent hashing via the ketama option, which moves only the keys that belong to the new server. They now restrict cache server changes to early Sunday mornings and plan database capacity around a cold cache.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in