SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Google Gemma 4 QAT Weights Deliver 2x Faster Decoding Than bf16 on AWS SageMaker

0
·5 views

A benchmark test on Amazon SageMaker compared Google's Gemma 4 E2B model in two formats — a full-size bf16 checkpoint and a quantization-aware trained (QAT) 4-bit weights version — running on the same NVIDIA L4 GPU instance. The QAT model decoded tokens at 105.1 tokens per second versus 51.3 for bf16, and handled 16 parallel requests at 1,077 tokens per second compared to 619 for the standard version. Both models produced identical results across 40 test questions, indicating no accuracy loss from quantization. The test was conducted on an AWS ml.g6.xlarge instance in the us-east-2 region using a vLLM container, with the only change between runs being the model checkpoint variable. The author also released a suite of Python MCP tools on GitHub to simplify deployment and management of vLLM-hosted models on SageMaker endpoints.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Fathom, Umami, or Clics: Choosing the Right Privacy-First Analytics Tool

Three privacy-focused website analytics tools — Fathom, Umami, and Clics — each serve different team needs depending on setup preferences and feature requirements. Fathom is a fully hosted, low-maintenance solution suited for teams wanting simple traffic and event reporting without managing any infrastructure. Umami offers both a managed cloud option and an open-source self-hosted version, making it ideal for teams that need deployment control along with advanced features like funnels, heatmaps, and session replays. Clics is a managed platform that combines visitor, campaign, conversion, and session reporting in one lightweight tool, winning on setup ease for teams needing deeper analytics without a self-hosted server. All three tools support cookieless measurement, but teams are advised to review full vendor terms and their broader tech stack to address all legal compliance questions.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Guide: Deploy Google Gemma 4 on Amazon SageMaker with vLLM and MCP Tools

A developer has published a step-by-step guide for deploying Google's Gemma 4 E2B model to an Amazon SageMaker real-time endpoint running on a single NVIDIA L4 GPU. The setup uses AWS's published vLLM container for SageMaker and relies entirely on plain AWS CLI commands, making each step executable manually or via an MCP server. A Python-based MCP server built on the MCP SDK 2.x exposes annotated tools for managing the deployment, from endpoint creation to deletion, with Claude Code acting as the MCP client. SageMaker handles infrastructure concerns such as container health checks, request routing, and CloudWatch logging, eliminating the need to manage instances or load balancers directly. The project code is publicly available on GitHub and requires Python 3.11 or newer, AWS CLI v2, and a SageMaker quota for a compatible ml.g6 instance type.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Trelix releases v3.2.2–v3.3.8, fixing broken Docker image and adding production CI gates

The Trelix project shipped thirteen tagged releases between August 26 and September 25, 2026, spanning versions v3.2.2 through v3.3.8, the current latest release on PyPI. The cycle began by patching two bugs in v3.2.2: a published Docker image missing the trelix-mcp binary entirely, and a console script that silently ignored all flags including --help and --version. Version v3.2.3 converted the manual artifact-testing process that uncovered those defects into permanent CI, adding end-to-end subprocess tests, fresh-venv installs, and a publish gate requiring a smoke-test build to pass first. The same release also fixed a SQL LIKE-wildcard escaping flaw where unescaped underscores and percent signs in directory names were misread as wildcards in path-filtered queries. While the code-level quality of new features improved across the span, documentation and changelogs repeatedly lagged behind — mismatched class names, unverified performance claims, and overstated guarantees emerged as the dominant failure pattern by the end of the period.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

WClickHouse adds built-in buffer manager to batch ClickHouse writes without Redis

WClickHouse, an open-source Python library developed by William Steve Rodríguez Villamizar, has introduced an integrated in-memory Buffer Manager that eliminates the need for external tools like Redis or Celery when writing streaming data to ClickHouse. The feature, highlighted on day three of the project's technical series, allows developers to enable buffering with a single parameter and automatically flushes data to ClickHouse once a configurable row threshold is reached. This approach prevents performance degradation caused by frequent single-row inserts at high frequency. The library supports Python versions 3.9 through 3.14 and is built on Apache Arrow and Pydantic v2. It is available on both GitHub and PyPI, with over 95% test coverage verified against real ClickHouse instances.

0 comments Read more at DEV Community