SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

How to Run a Local LLM on Apple Silicon Using MLX or llama.cpp

0
·1 views

Mac users with M1 through M4 chips can run large language models entirely on-device using two leading tools: Apple's MLX framework and the community-built llama.cpp engine. MLX is installable as a Python package and can load a 4-bit quantized 3B model in minutes, requiring no API keys or cloud connectivity. The two tools differ in scope — MLX is optimized for Apple's unified memory architecture, while llama.cpp supports a broader range of hardware including Linux and Windows systems. Both frameworks support quantized model formats and offer OpenAI-compatible server modes for application development. Running models locally eliminates per-token costs and keeps user prompts private, with the setup working fully offline.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer builds static site tracking 14 years of meals eaten across Mexico

A developer created 'Sobremesa', a single static HTML page documenting six meals eaten across six Mexican cities over fourteen years. The project is built around the Spanish concept of 'sobremesa' — the time spent lingering at a table after a meal ends — and tracks how long the author and his wife stayed at each table rather than rating the food itself. The six cities covered include Tijuana, Rosarito, Ensenada, Guadalajara, Mexico City, and Oaxaca, totalling nine hours and thirty minutes across all tables. Each entry includes the dish, location, a verified food fact, and a personal note, while a built-in form lets visitors generate a downloadable card for their own memorable meal. The site uses no framework, no build step, no tracking, and no cookies, and was submitted to the DEV Frontend Challenge under the Comfort Food Edition prompt.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Three coding tools already blend local and cloud AI models — but smart routing remains unsolved

A question on r/LocalLLaMA about running local and cloud models together in a single coding workflow prompted a detailed investigation into what tools actually support this. Three open-source coding agents — Cline, Aider, and Continue — already enable hybrid model use by assigning different models to distinct roles such as planning, editing, autocomplete, and commit messages. Unlike proxy solutions such as LiteLLM, these tools allow a local model and a cloud model to divide labour within the same task session. However, none of these harnesses dynamically select a model based on the specific requirements of a subtask at runtime — that capability does not yet exist in any shipping tool. The gap persists because the context needed to make such decisions is generated during planning but is never passed down to the model-selection layer.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds Open Sandbox to Test Whether AI Coding Agents Respect Security Boundaries

A software developer has released a lightweight, runnable "canary harness" designed to test whether AI coding agents stay within safe operational boundaries during routine tasks. The tool plants four types of traps inside a disposable sandbox directory, including a fake API key file, git hooks, a local network endpoint, and a prompt-injection comment in a README, then monitors whether the agent touches any of them. Inspiration came after the developer observed an AI agent unexpectedly reading a .env file and exposing a database URL while performing a simple refactoring task. The harness uses standard Linux tools such as auditd, netcat, and git to log file access and network calls without requiring specialized infrastructure. Repeated runs with varied file names and injection phrasings are recommended to assess behavioral consistency rather than drawing conclusions from a single test.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Publishes Reusable Test Harness to Probe AI Agent Tool Boundaries

A software developer has shared a repeatable Python-based test harness designed to expose security weaknesses in AI agents before they reach production. The tool targets four common failure modes: scope creep, confused deputy attacks, escalation by composition, and silent refusal failures, where an agent claims to decline a request but executes it anyway. A key design principle of the harness is that it evaluates only the structured tool-call trace rather than the agent's natural-language output, which the author argues is inherently untrustworthy. The harness uses a combination of hand-written seed cases and an LLM-generated adversarial input set to stress-test tool boundary rules defined in a schema. It is built to run in CI pipelines and is compatible with free-tier models to avoid unexpected costs during red-team testing.

0 comments Read more at DEV Community
How to Run a Local LLM on Apple Silicon Using MLX or llama.cpp · ShortSingh