Why Custom AI Chips Like TPUs Beat GPUs at Scale but Struggle With Low-Batch Tasks

·4 views

Purpose-built AI accelerators such as Google's TPUs differ fundamentally from GPUs by sacrificing general programmability to maximise matrix multiplication efficiency, freeing up silicon area for more compute units and on-chip memory. Their core architectural feature, the systolic array, reuses data across a grid of multiply-accumulate cells, dramatically improving the ratio of floating-point operations to memory bandwidth. However, this specialisation comes with a structural weakness: small or irregularly shaped matrix operations leave processing cells idle, unlike GPUs which can repurpose their lanes more flexibly. The efficiency gap is widest during high-throughput batch inference and narrowest at batch size one, such as single-token generation in language models. Because the economics of custom chip design require massive scale to recover fixed costs, this silicon typically exists as cloud capacity rather than a purchasable component, meaning buyers are effectively evaluating a hosted service.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

LangGraph Tutorial Part 5: Agent Learns to Balance Shared Driver Pools Across Two Zones

A developer tutorial series on building a ride-share zone-balancing AI agent concludes with its fifth and final installment, focusing on coordinating two zones simultaneously rather than evaluating each in isolation. Previous parts assumed an unlimited driver pool, but the new installment addresses the reality that regional driver supplies are finite and shared between zones. Each zone gains two additional supply channels: a local dormant driver pool and a cross-zone pull request that draws from a neighboring zone's surplus. A human approval step is triggered only when a zone's cross-zone request exceeds what the adjacent zone can comfortably spare, keeping most automated cycles uninterrupted. The tutorial also explains key architectural decisions in LangGraph, including why each zone's logic is wrapped as a compiled sub-graph to ensure reliable parallel fan-in behavior.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

LangGraph Ride-Share Agent Gets Human Checkpoints for Critical Decisions

A developer tutorial series on building a ride-share zone-balancing AI agent has reached Part 4, focusing on adding human oversight to the LangGraph-powered system. The update introduces conditional interrupts that pause agent execution at two specific points: once on the first cycle to let a human verify raw zone data, and again whenever a deficit is classified as critical before a costly policy is applied. All other severity levels — mild, moderate, balanced, and surplus — continue to run fully autonomously as before. The implementation uses LangGraph's interrupt() function inside plain Python nodes, with execution resuming only after a human passes a response via Command(resume=...). A noted technical caveat warns that passing a falsy value like an empty dict to Command(resume=) will cause LangGraph to re-trigger the interrupt instead of continuing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

LangGraph Tutorial Part 3: Adding Persistent Memory to a Ride-Share Balancing Agent

A developer tutorial series on DEV Community has published its third installment, detailing how to give a LangGraph-based ride-share zone-balancing agent persistent memory across multiple decision cycles. The update introduces LangGraph's MemorySaver and a thread_id so that each zone maintains a continuous state across repeated 15-minute operational windows, rather than resetting on every call. Two new non-LLM nodes — start_cycle and apply_scheduled_conditions — are added to manage clock advancement and execute conditions scheduled by earlier operator notes. The agent's core decision logic remains unchanged from Part 1, while the AgentState is extended with four new fields to track cycle number, clock anchor, run history, and scheduled conditions. The series is planned for five parts total, with this instalment focused purely on memory architecture rather than changes to balancing or explanation logic.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Dev Tutorial: Using LLMs Sparingly in a Ride-Share Zone-Balancing Agent

A developer tutorial series on DEV Community details how to integrate a large language model into a ride-share zone-balancing agent built with LangGraph, now in its second of five parts. The core insight is that LLMs are used for only two narrowly defined tasks: parsing free-text operational notes into structured flags, and generating plain-English explanations for human reviewers. All policy selection logic remains purely deterministic Python code carried over from Part 1, deliberately kept away from the LLM to avoid unreliable outputs. The author explains that an earlier ReAct-loop approach, where the LLM selected policies itself, failed because language models generate plausible text rather than reliably computing numeric comparisons. Five local LLM models are benchmarked side by side to evaluate which performs adequately even within these limited, well-defined roles.

0 comments Read more at DEV Community