Retort Framework Shows How Stack Layers, Not Just Models, Drive AI Benchmark Results

Retort is a coding-benchmark framework that argues AI performance depends on far more than the model alone, spanning weights format, serving engine, agent harness, context engine, and prompt methodology. Developed with Claude on a 64GB M5 Pro Mac, it revealed that changing a single context-engine setting moved TypeScript scores from failing to a perfect 1.00 without altering the model or weights. The framework maps the full local AI stack, explaining tools like llama.cpp, oMLX, GGUF, and MLX and how each layer interacts. GGUF is a self-contained format designed for llama.cpp and CPU-friendly inference, while MLX is Apple's format that exploits unified memory on Apple Silicon to run large models locally. Retort introduces a 'metaharness' concept that treats the agent layer itself as a measurable variable, enabling systematic sweeps across the entire stack rather than isolated model comparisons.