Apple Silicon VMs Achieve Up to 16x Faster LLM Inference via GPU Passthrough

·1 views

A technical blog post published on GitHub details how macOS virtual machines running on Apple Silicon can dramatically accelerate large language model inference. The approach leverages GPU passthrough within VMs to run Llama.cpp, yielding performance gains of 11 to 16 times over CPU-bound alternatives. The findings were shared by the team behind the open-source project 'cua' on their repository. The post highlights that Apple Silicon's unified memory architecture makes such GPU access within virtualized environments particularly effective for AI workloads.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Dev Series Finale: Real Bugs Found Building an AI Support-Ticket Agent From Scratch

A developer has concluded a seven-part series documenting the construction of a support-ticket AI agent without using any framework, with the final installment focusing on real bugs uncovered during evaluation. Key issues included flawed trajectory-matching logic that caused false failures, a lack of retry logic that let a single 503 error crash the entire eval run, and a 'latest' model alias silently switching to a version with far stricter API quotas. A more serious bug revealed the language model falsely claimed a refund had been issued without ever calling the relevant tool, which was fixed through both prompt-level instructions and a hard code-level integrity check. The author also addresses the multi-agent question directly, noting that a coordinator-plus-worker architecture costs 3–8x more in tokens and latency per run and should only be adopted when specific conditions are met.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Dev Tutorial: How to Add Observability and Drift Detection to AI Support Agents

A developer tutorial series building an AI support-ticket agent from scratch has reached its sixth installment, this time focusing on observability. The guide explains how production traffic differs from offline evaluation sets, arguing that real-time monitoring is essential to catch unexpected ticket types the agent was never tested on. The implementation logs structured trace data for every tool call, including hashed arguments, truncated result summaries, latency, and token usage, while avoiding storage of raw sensitive values. A lightweight CLI formatting layer renders traces in human-readable form during live runs, with color output that automatically disables in non-terminal environments. The tutorial outlines three observability layers built on top of per-step tracing: detailed trace logs, aggregate online metrics, and drift detection.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Shows Why AI Agent Safety Rules Belong in Code, Not Prompts

A software developer building a support-ticket AI agent from scratch discovered that the model falsely reported proposing a refund without ever calling the required tool, highlighting a critical reliability gap. To address this, the developer implemented a policy enforcement layer in TypeScript that controls tool access, rate limits, and approval requirements entirely outside the language model's influence. High-stakes actions like issuing refunds and sending emails require explicit human approval before execution, ensuring the model cannot bypass these checks through its generated text. Regex-based escalation patterns also screen incoming tickets for legal threats or fraud indicators before the AI is even invoked, routing flagged cases directly to human agents. The findings make a concrete case that safety guardrails enforced in code are fundamentally more reliable than instructions embedded in prompts alone.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds AI Support Agent in ~100 Lines of TypeScript Without Any Framework

Software developer Akash Pal has published Part 4 of a series documenting how to build a support-ticket AI agent from scratch, without relying on any external orchestration framework. The core agent loop, written in TypeScript, works by repeatedly calling a language model, checking whether it requests a tool, executing that tool if needed, and returning a final answer when no further tool calls are made. The entire loop is implemented as a simple for-loop with an if-statement, deliberately avoiding abstractions like state machines or graph-based orchestrators. A structured system prompt acts as a behavioral specification, enforcing policies such as never issuing a refund without a prior eligibility check and always prefixing responses with a parseable outcome label. Pal argues that understanding the loop in plain terms before adopting a framework leads to more transparent and debuggable agent design.

0 comments Read more at DEV Community