Gemma 4 QAT Weights Deliver 2x Faster Decoding Than bf16 on AWS SageMaker L4

·3 views

A developer benchmarked Google's Gemma 4 E2B model in two formats — full-precision bf16 and a quantization-aware trained (QAT) 4-bit weights checkpoint — on an Amazon SageMaker real-time endpoint powered by a single NVIDIA L4 GPU. The QAT variant decoded tokens at 105.1 tokens per second compared to 51.3 for bf16, and handled 16 parallel requests at 1,077 tokens per second versus 619 for the standard model. Both checkpoints produced identical results across 40 test questions, suggesting no accuracy loss from quantization. The test was conducted on an ml.g6.xlarge instance in the us-east-2 region using AWS's vLLM SageMaker container, with the only configuration change being the model checkpoint variable. A companion suite of Python MCP tools was also built to streamline deployment and management of the vLLM-hosted endpoint.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How one developer translated a TV tracker app into 46 languages for free

A solo developer behind dizi.jpg, a TV show and movie tracking app, needed to translate a large catalogue of descriptions into 46 languages but ruled out commercial translation APIs due to prohibitive per-character costs. Instead, they built an offline batch translation pipeline using open-source tools Argos Translate and CTranslate2, running translations on their own computer and storing results in a PostgreSQL database. This approach eliminates per-request translation costs and latency, while also allowing search engines to index fully translated pages for each language. The trade-offs include lower output quality compared to commercial systems, potential errors when no direct language-pair model exists, and a lag before new content appears in translated languages. The developer recommends this method to other solo or small-team content-heavy projects seeking multilingual support without ongoing API bills.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Study Finds Auth, Access Rules and Webhooks Are Top AI App Launch Failures

A structured analysis of 4,469 public builder posts identified 59 verified cases in the past year where AI-powered apps broke at or after launch. The most common failure points were user authentication flows, hand-written row-level security rules, file storage, and payment webhooks. Sign-in issues ranged from broken email links and misconfigured cookies to token refresh race conditions that logged paying users out mid-session. Access control failures were particularly severe, with misconfigured or missing row-level security policies allowing users to read or modify other customers' data. Researchers recommend testing full auth flows on production domains, enforcing deny-by-default database policies, and logging detailed failure reasons rather than generic error codes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Google Adds Live Voice Chat, Mobile Audio Recorder to Gemini Notebook

Google announced on September 15, 2026, a set of updates to Gemini Notebook that introduce real-time voice conversations, a mobile audio recorder, and new interactive learning outputs. The voice conversation feature allows users to speak directly with their notebooks in nearly 100 languages across both the mobile app and web. A mobile audio recorder is also being introduced to help users capture lectures and spontaneous thoughts while away from a desk, with its rollout beginning the week after the announcement. New learning-focused outputs — including quizzes, flashcards, and Short Video Overviews — are being added alongside the existing Audio Overview feature. The changes reflect Google's broader effort to make Gemini Notebook more interactive and to consolidate the product within its Gemini ecosystem.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Agents vs Traditional Automation: Why the Security Gap Matters

Unlike traditional automation, where every execution path is pre-coded and reviewed by engineers, AI agents dynamically choose their own next steps, tools, and sequences at runtime. This shift moves control from deterministic code into a model-driven decision loop, fundamentally expanding the attack surface to include the agent's tool set, permissions, and memory. Security bodies including NIST and OWASP have flagged AI agents as vulnerable to 'agent hijacking', where prompt injection can trigger harmful actions rather than just bad text output. Anthropic's December 2024 guidance formally distinguishes workflows, which follow predefined code paths, from agents, which direct their own processes dynamically. Security teams are advised to treat the model as an untrusted decision-maker and keep identity, authorization, argument validation, and logging deterministic and outside the model's control.

0 comments Read more at DEV Community