Tollgate v0.2.3 Adds Long-Context Pricing Tiers to Prevent LLM Budget Surprises

·2 views

Tollgate, an open-source LLM gateway that reserves and settles costs per request, has released version 0.2.3 with improved handling of long-context pricing tiers. Some AI providers, including Google for Gemini 2.5 Pro on Vertex AI, charge a higher rate on the entire request once a prompt crosses a token threshold, not just on the excess tokens. A proxy storing a single flat rate per token class will silently under-charge such requests, potentially causing invoice discrepancies worth thousands of dollars. The update introduces configurable per-model thresholds and rate multiples, applied consistently to both cost reservations and final settlements to prevent drift. Until a tier is explicitly configured, Tollgate logs oversized prompts as possible under-charges rather than applying an assumed uplift.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Graph Databases Offer Agents a Better Way to Answer Relational Questions

Most AI agent retrieval systems rely on vector indexes, which are well-suited for similarity-based queries but cannot handle questions about relationships or dependencies between entities. Graph databases address this gap by modeling data as nodes and edges, allowing agents to traverse connections of unknown depth without complex recursive queries. A graph query language like Cypher lets developers express multi-hop traversals concisely, making it easier to answer questions such as what services depend on a given component. Published benchmarks for one such graph database cite sub-millisecond two-hop query latency and high read throughput, with the vendor encouraging readers to review the full methodology before drawing conclusions. The post invites developers to share their experiences with graph-based retrieval in production, particularly around token efficiency compared to traditional vector or relational approaches.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

NexusOS: Open-Source AI Assistant That Controls Your PC via Voice or Text

A developer has released NexusOS, an open-source, locally-run AI environment designed to control a computer through voice or text commands. The system can automate browser tasks, manage files, move the mouse, and execute multi-step routines without keyboard input. It uses PyAutoGUI for desktop control and Playwright for browser automation, while also supporting smart home integration via MQTT and Home Assistant. NexusOS retains context across sessions using vector embeddings, giving it persistent memory of past interactions. The project is available on GitHub and supports extensibility through a Python-based plugin system.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

PureScript Pitched as Middle Ground Between AI Code Generation and Manual Mastery

A developer essay published on DEV Community argues that the debate over AI-assisted coding has split programmers into two camps: those who fully delegate writing to models and those who insist on manual control over every line. The author proposes that PureScript, a pure statically typed functional language similar to Haskell, could bridge this divide by keeping code structure closely aligned with business logic and readable as a specification. Using a refund-decision example, the piece demonstrates how PureScript type definitions can communicate intent clearly enough for both humans and AI models to reason about before any implementation details are written. The author draws a parallel between traditional compilers and modern AI code generators, suggesting both perform the same conceptual task of translating high-level ideas into lower-level instructions. The conclusion is that expressive, strongly typed languages may allow developers to confidently delegate more code generation while retaining meaningful oversight and understanding.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Fit Long-Context LLMs Into 16 GB GPU VRAM Using KV Cache Tuning

Running large language models with long context windows on 16 GB GPUs is constrained not by advertised limits but by physical VRAM shared among weights, KV cache, and compute buffers. The KV cache grows with every active token, meaning a configuration that appears stable at startup can slow down, spill into system memory, or crash during large prefills. A practical VRAM budgeting approach covers context sizes from 32K to 128K tokens, with guidance for tools like llama.cpp, vLLM, and Ollama. Cache size depends heavily on a model's attention architecture — such as grouped-query or multi-head latent attention — rather than its parameter count alone. Experts recommend starting with a single sequence, Flash Attention, and an 8-bit KV cache before attempting more aggressive optimizations, noting that a stable 64K setup is generally more reliable than a 128K one operating near out-of-memory limits.

0 comments Read more at DEV Community