Session Management II: Adding Context Compaction to a Basic AI Agent

·2 views

This tutorial continues a series on building a basic AI agent from scratch, focusing on session management. Long-running sessions eventually exceed the model's context window and become slow and costly, so the author adds context compaction. The harness estimates token usage before each request and automatically compacts once it passes 80% of the token budget, or on demand via a manual command. Compaction splits the conversation at the latest user-turn boundary so tool calls and results stay paired, then summarizes older turns with a separate, low-temperature LLM call while the full transcript remains on disk. The summary is inserted as a clearly labeled untrusted-data user message to reduce prompt-injection risk, and the summarization call's usage is tracked in the session's cost estimate.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Roku Moves Developer Repositories from GitHub to GitLab

Roku has relocated its developer code repositories from GitHub to GitLab, according to a company blog post. The announcement was shared on Hacker News, where it gathered 14 points and one comment. The source material provides few additional details about the timing or the reasons behind the switch. Developers who rely on Roku's repositories may need to update their links and workflows to reflect the new hosting location.

0 comments Read more at Hacker News

ProgrammingHacker News ·

Neural chip decodes interactions between brain regions

Researchers have developed a neural chip capable of decoding how different regions of the brain interact with one another. The work was reported by EPFL, the Swiss Federal Institute of Technology in Lausanne, as described in a news release. The chip is designed to read signals passing between brain areas rather than only activity within a single region. The source material available for this item is limited, so details on the study's methods, scale, and findings are not included here.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

DeepSeek-Linked Paper Proposes Engram, a Lookup Memory Module for Transformers

A new paper from a DeepSeek-linked team argues that large language models waste computation by re-deriving static facts on every query, and proposes Engram, a module that retrieves stored knowledge through a direct lookup. Scaled to 27 billion parameters, the authors report that it outperforms a mixture-of-experts baseline with the same parameter and compute budget across the benchmarks they tested. The largest gains appear on reasoning and code tasks rather than pure knowledge tests. The authors frame the design as a budgeting problem, finding a U-shaped trade-off between neural computation and static memory, with an optimal mix in between. Their Engram-27B model reallocates parameters from expert modules into a 5.7 billion parameter memory, which they say frees the network's early layers and attention capacity for harder work.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Guide: Running a 4B Qwen3 Chat Model Locally on Windows with a Small NVIDIA GPU

A DEV Community guide explains how to run the 4-billion-parameter Qwen3-4B chat model on a Windows PC with an NVIDIA RTX 30, 40, or 50 series GPU, using a prebuilt zip from the author's open-source Pingala-8 project. The only required install is a recent NVIDIA driver, since the package bundles the CUDA runtime files and avoids the full CUDA Toolkit and any compilation. Users extract the zip and run a batch script that downloads the roughly 1.24 GB model from Hugging Face on first launch, then starts a local llama.cpp server and opens a browser-based chat page at 127.0.0.1:8080. The author reports about 200 tokens per second on an RTX 5070 Ti Laptop GPU and notes that the setup requires roughly 5 GB of free disk space and an internet connection for the initial download.

0 comments Read more at DEV Community