How to Run ML Models On-Device in Android Apps at 35ms Inference

·3 views

A software engineer outlines a complete pipeline for integrating machine learning models into Android apps using TensorFlow Lite, drawing on three years of real-world client work. The approach was originally driven by a health-tech startup that needed on-device skin-screening to avoid sending sensitive medical images to the cloud and stay GDPR-compliant. An early attempt failed due to unquantized models running on the main thread, causing bloated APK sizes, UI freezes, and poor battery performance. The article details how converting models from float32 to int8 via post-training quantization can reduce model size by 75 percent and significantly speed up inference. Key recommendations include choosing between on-device, cloud, or hybrid inference based on privacy needs, and keeping the model conversion pipeline in CI to prevent versioning errors.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

MeshDrive 2.0 gives AI agents isolated local file storage via MCP on Linux

MeshDrive 2.0 is a local-first Linux storage tool built on JuiceFS and SQLite that provides AI agents with controlled access to a dedicated file system. Released by developer Hardik94 on GitHub, version 2.0 adds a Model Context Protocol (MCP) server enabling AI clients such as Cursor, Claude Code, and Hermes to read, write, and list files through a defined set of tools. Agents are restricted to an isolated root directory by default, preventing them from accessing files outside the designated storage area. The free tier requires no cloud connectivity, keeping all data on local disks, and includes a web-based file browser accessible at localhost. Installation is supported on Ubuntu and Debian amd64 systems running Python 3.10 or later with FUSE support.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Best Practices for Defining AI Agent Skills: Structure, Naming, and Invocation

AI agent skills are used to automate repetitive, precisely describable processes, and must be carefully structured to minimize the risk of a model generating incorrect outputs. Skills can be invoked either explicitly by a developer or implicitly by the agent itself, and each one loaded into a session consumes context, making selective use important. Each skill must reside in its own dedicated directory, with the directory name matching the skill's name, and must contain a SKILL.md file holding its definition. The skill's YAML header requires at minimum a name — formatted in lowercase kebab-case between 1 and 64 characters — and a description of up to 1,024 characters specific enough for the agent to match it to user intent. Optional fields such as license, version, and context can further refine how a skill is managed, deployed, and legally used across different environments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Redis Sliding Windows and Token Quotas Protect LLM API Spending

Rate limiting LLM APIs differs fundamentally from conventional APIs because a single model call carries real monetary cost, and agent loops can fire hundreds of requests autonomously. Developers must track both request counts and token usage separately, enforcing whichever limit binds first, since a one-line prompt and a 200-page document summarisation both count as one request but differ vastly in cost. Tenant identifiers used in rate-limit keys must be sourced from authenticated sessions, never from the request body, to prevent callers from manipulating or exhausting other tenants' quotas. Fixed time windows expose a known boundary exploit where a full quota can be consumed twice in rapid succession, making Redis sorted-set sliding windows a more reliable alternative when executed atomically via Lua scripts. Because actual token costs are only known after a model responds, the recommended approach is to reserve an estimated token count before the call and reconcile the difference once the response returns.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

tanstack-fetch: A Typed Fetch Client Designed to Simplify TanStack Query HTTP Layer

A developer has released tanstack-fetch, an unofficial, typed fetch client built specifically to complement TanStack Query in JavaScript and TypeScript applications. The library addresses common pain points that arise as apps scale, including authentication, typed error handling, request cancellation, retries, and server-sent events. Unlike native fetch, which does not throw on HTTP error status codes like 404 or 500, tanstack-fetch automatically converts HTTP failures into structured FetchError objects that integrate cleanly with TanStack Query's error state. The package can be configured with a base URL, token retrieval logic, and status-specific handlers, reducing repetitive boilerplate across query functions. It requires Node 18 or later in server environments and is available via npm alongside @tanstack/react-query.

0 comments Read more at DEV Community