Why Experienced Engineers Choose Data Structures Last, Not First

·4 views

A software engineering guide argues that selecting the right data structure should always begin with understanding the business problem, not the implementation. Experienced engineers first identify what behavior a system must support before deciding on tools like HashMaps, Queues, or Graphs. The article maps common business questions to their naturally fitting data structures, such as using a Heap when prioritization is needed or a Trie when partial-input search is required. It warns that jumping straight to implementation choices is a common beginner mistake that leads to poorly designed components. The core takeaway is that data structures are not independent decisions but logical consequences of clearly understanding business behavior.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Filter First, Allocate Later: Why Selective Attention Beats Truncation

A technical analysis argues that when managing token budgets in AI systems, filtering irrelevant content before allocating space is far more effective than truncating documents to fit. Using a 30,000-token budget example, the piece shows that filtering down to three relevant complete documents outperforms cramming ten truncated ones, while using the same token count. The core principle is to prefer fewer whole documents over many partial ones, since truncated documents lose their conclusions and become unreliable. A proper filter requires three components: a scoring function, a relevance threshold, and a cap on the number of admitted documents. The article also provides a cost-benefit formula to determine when running a filter model is economically justified compared to simply passing all candidates to the main model.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

When to Use AI Context Compression — and When Caching Beats It

Managing large language model context windows efficiently requires choosing between several techniques, including deletion, retrieval, caching, and compression. Lossless cleanup — stripping markup, minifying structured data, and deduplicating chunks — can reduce token counts by 30–60% before any model-based method is applied. Model-based approaches such as extractive selection, abstractive summarisation, and token pruning (notably Microsoft Research's LLMLingua) offer higher compression ratios but carry costs and quality trade-offs. A cost comparison shows that caching a stable 20,000-token context is both cheaper and lossless compared to compressing it, making compression most justified when context changes on every request or must be shared across models. The recommended decision order is: delete useless tokens first, retrieve instead of stuffing context, cache what is stable, and only then compress what remains.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Prompt Prefix Order Determines Whether AI Cache Hits Ever Occur

Prompt caching in AI systems works strictly on prefix matching — a provider can only reuse computed state up to the first token that differs between two requests. This means content that repeats but appears in a different position offers no cost savings whatsoever, making block ordering the single most critical factor in cache efficiency. Developers are advised to arrange prompt sections from most stable to least stable, placing system prompts and static references first, followed by session data, conversation history, and finally per-request content like retrieved documents. Common mistakes that silently break caching include injecting timestamps, unstable JSON serialisation, per-user data at the top of prompts, and unique request IDs added by middleware. Conversation history is cacheable despite growing each turn, because new messages are appended and the earlier prefix remains unchanged — but only if no volatile content is placed after it.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Properly Allocate Token Budgets Across an LLM Context Window

Managing a large language model's context window requires deliberately dividing a fixed token limit among competing content blocks — system prompts, history, and retrieved documents — before any request is assembled. A key arithmetic mistake is filling the window to capacity without reserving space for the model's output, which causes requests to fail. Each content block should be assigned a floor (minimum useful size), a desired size, and a priority, so that when space is tight, lower-priority blocks are dropped entirely rather than all blocks being uniformly truncated. Fixed elements like system prompts and tool schemas must be allocated first since they cannot be scaled down, while elastic blocks like chat history share whatever space remains. A small safety margin of two to three percent should also be set aside to account for token-count discrepancies introduced by chat templates and role markers during server-side re-serialization.

0 comments Read more at DEV Community