Cactus fine-tunes Gemma 4 with a confidence probe to cut cloud AI costs

·1 views

Startup Cactus has post-trained Google's Gemma 4 E2B model with a lightweight 68,000-parameter probe layer that predicts how likely the model is to be wrong on any given response. Each output is assigned a confidence score between 0 and 1, allowing developers to process high-confidence queries entirely on-device and route only uncertain ones to a larger cloud model. By sending just 15–55% of queries to Gemini Flash-Lite depending on the task, the hybrid system matches the cloud model's benchmark performance across text, vision, and audio tasks. The probe achieves an average AUROC of 0.814 across 12 hold-out benchmarks, significantly outperforming token entropy heuristics which scored 0.549. Cactus has released the model weights on HuggingFace and the code under an MIT license, with support for Transformers, MLX, and Llama.cpp available now.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Developers Should Treat AI Prompts Like Code, Not Conversation

A developer building a local AI agent found that adding more instructions to a lengthy system prompt caused the model to forget earlier rules, highlighting the limits of so-called mega-prompts. The article argues that mixing instructions and data into unstructured text creates the AI equivalent of spaghetti code, and advocates for a structured approach called Context Engineering. Using XML tags to clearly separate directives from data — a technique documented by Anthropic and applicable to most models — can significantly improve output reliability. Providing concrete good-versus-bad examples within this structure is described as more effective and token-efficient than writing lengthy theoretical instructions. The author draws a parallel to software development best practices, urging teams to treat prompts like configuration files and break them into modular, maintainable components.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Talon Audit Launches Privacy-First Platform for Full-Spectrum Website Analysis

A developer has launched Talon Audit, a SaaS platform designed to assess websites across four interconnected domains: security exposure, integrity, user experience, and conversion risk. Unlike single-focus audit tools, it aims to reflect how visitors actually experience a website by combining these areas into one unified workflow. The platform follows a five-stage process — detect, explain, fix, verify, and monitor — providing evidence-backed findings to help teams prioritize action. Talon performs only passive, low-impact analysis and requires users to confirm site ownership or authorization before testing, with customer reports kept private. AI tools including Codex and Claude were used to accelerate development, though core product decisions were made by the human developer.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why engineers are moving Firestore from source of truth to read-only cache

A software engineer has shared an architectural pattern that repositions Firestore as a read-optimized cache rather than the primary database for production applications. In this approach, Cloud SQL serves as the authoritative source of truth, with all writes routed through a Next.js and Cloud Run backend that updates both Cloud SQL and Firestore in the same write path. A Cloud Run reconciliation job, queued before each write begins, automatically detects and repairs any inconsistency between the two stores if a write fails midway. The setup preserves Firestore's strengths — real-time push via onSnapshot and flexible data modeling — while offloading transactional consistency and complex queries to SQL. The author argues that for services intended to grow over years, treating Firestore as a rebuildable cache from the outset is far cheaper than migrating away from it later.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Serverless Computing: Key Strengths, Limitations, and When to Use It

Serverless computing offers significant advantages for event-driven, intermittent workloads, allowing developers to avoid managing infrastructure and pay only for actual execution time. It scales automatically during traffic spikes and enables rapid API deployment, making it well-suited for webhooks, data transformation, and similar single-purpose tasks. However, the model has notable drawbacks, including execution time limits, cold-start latency, and the inherent stateless design that requires additional services for persistent connections or local state. For applications with constant, high-volume traffic, traditional provisioned servers can prove more cost-effective than per-invocation serverless pricing. Experts recommend carefully evaluating workload characteristics before adopting serverless, as misapplication can introduce new complexity rather than reduce it.

0 comments Read more at DEV Community