Sleep-Time Compute Cuts AI Agent Latency by Pre-Answering Queries While Idle

·5 views

Researchers at Letta (Lin et al., 2025) have proposed a technique called sleep-time compute, which shifts AI inference work to idle periods between user sessions rather than processing everything on demand. A background worker pre-answers queries likely to be asked again and compresses standing context into dense summaries, so the system can serve warm, instant responses when users return. Each pre-computed answer is tagged with the source version it was derived from, and a freshness check ensures stale answers are discarded and recomputed rather than served to users. In a demonstration across 400 queries — 70% predictable and 30% novel — foreground latency dropped by 57% and foreground cost fell by over half, while novel queries were still handled live. The approach is particularly suited to workflows where users repeatedly query the same documents or codebases, since the underlying context changes infrequently between sessions.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Google Renames NotebookLM to Gemini Notebook, Adds Thinking Steps and Workspace Sync

Google has rebranded and expanded its notebook research tool as Gemini Notebook, integrating it more deeply with the Gemini app, Google Search, and Google Workspace. A key new feature allows eligible subscribers to view the model's thinking steps during chats and notebook operations, offering greater transparency into how responses are generated. Google has also added in-notebook code execution while retaining existing features like citation-backed responses, Audio Overviews, and Mind Maps. The rollout is phased, with initial access for Google AI Ultra and Workspace AI Ultra Access users, followed by broader availability for Pro subscribers on the web. Google cautions that the visible thinking steps are experimental and should be treated as an inspection aid rather than confirmation of output accuracy.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Measure True LAN Speed Between Devices Using iperf3

iperf3 is an open-source, client-server network testing tool maintained by ESnet that measures actual throughput between two devices on a local network, something internet speed tests cannot do. Unlike services such as Speedtest, iperf3 isolates a specific link — such as phone to NAS or mesh node to access point — by controlling both endpoints directly. One device runs as a server using 'iperf3 -s', while the client connects and pushes traffic for a set duration, reporting sustained throughput, retransmits, jitter, and packet loss. The tool is available via standard package managers on Linux and macOS, and native apps exist for iPhone, iPad, and Android to allow mobile testing across different rooms or locations. Walking through a building while running tests reveals real-world Wi-Fi performance gaps that signal-strength indicators alone cannot expose.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Neural Networks Process Text: Tokenization, Matrix Math, and Deep Learning Explained

Natural language processing models cannot work with raw text directly, so words are first converted into numerical token IDs before being fed into a neural network. Each layer of the network takes an input vector, multiplies it by a matrix of learned weights, adds a bias, and passes the result through a non-linear activation function like ReLU. Stacking many such layers — the basis of deep learning — allows networks to build progressively richer representations of the original input. Recurrent neural networks extend this by combining the current input vector with a running hidden state at every timestep, though the fixed size of that state limits how much information can be retained over long sequences. This matrix-based structure is also why GPUs are central to deep learning, as they are purpose-built to execute large-scale matrix multiplications efficiently.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AbilityGuard v1.1 adds persistent logs, advanced filters, and watchlist alerts

The WordPress plugin AbilityGuard has released version 1.1, significantly expanding its monitoring capabilities for the WordPress Abilities API. The update replaces the previous rolling 100-entry log cap with configurable or unlimited log retention, ensuring audit history is no longer silently lost. Advanced filtering options — covering status, risk level, user ID, date range, and more — now work independently of each other, and users can save filter combinations for reuse. Filtered log views can be exported directly to CSV, making the data easier to use in compliance reports or security reviews. A new sensitive ability watchlist lets users flag specific high-risk abilities for closer attention, shifting the tool from a passive dashboard to an active monitoring system.

0 comments Read more at DEV Community