Developer Builds Open-Source Scheduler to Juggle Multiple LLMs on a 12GB GPU

·1 views

A developer running an NVIDIA RTX 3060 with 12GB VRAM found that while modern 7B–12B language models run well on the hardware, loading multiple specialized models simultaneously was impossible. The core challenge became a scheduling problem: deciding which model occupies the GPU at any given time and managing the unload-and-reload cycle efficiently. To solve this, the developer built an open-source tool called GGUF Switchboard, which exposes a single OpenAI-compatible API endpoint while handling model lifecycle management underneath. The scheduler supports both GGUF and SafeTensors formats and can route requests to either llama.cpp or vLLM backends depending on the hardware. Model swaps on the RTX 3060 take only a few seconds, and the scheduler batches consecutive requests for the same model to minimize switching overhead.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why navigator.clipboard.writeText() Silently Fails When Users Switch Tabs

A subtle browser behavior can cause clipboard copy functions to fail without any visible error when users switch tabs or windows mid-operation. The Async Clipboard API requires the document to have active focus at the exact moment writeText() is called, not merely when the user first clicked. If an awaited operation — such as a fetch request — creates a timing gap between the click and the clipboard call, focus may have shifted, causing the browser to silently reject the promise with a NotAllowedError. Common fixes like loading spinners, try/catch blocks, or retries do not resolve the issue because they fail to address the root cause of lost document focus. The recommended solution is to pass a promise directly into ClipboardItem so the clipboard slot is claimed during the original click gesture, while the data resolves asynchronously afterward.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

GPT4All Lets You Run AI Language Models Locally Without Internet Access

GPT4All is a free, open-source desktop application that allows users to run large language models locally on Windows, macOS, and Linux without an internet connection. It supports GGUF-format models from sources like Hugging Face and can also connect to API-based providers such as OpenAI, Groq, and Mistral. The application can be installed on Ubuntu and Debian-based systems via an installation binary or Flatpak, while Windows and macOS users have dedicated installers available. GPT4All also supports retrieval-augmented generation (RAG), enabling users to chat with their own local documents. Additionally, it includes a built-in API server that can be secured and made remotely accessible via HTTPS behind a reverse proxy.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

NVIDIA Dynamo and vLLM Enable Scalable LLM Inference Across Multi-GPU Setups

NVIDIA Dynamo is an open-source inference framework designed to deploy large-scale generative AI models across multi-node, multi-GPU environments with high throughput and low latency. It supports both aggregated serving for single-GPU configurations and disaggregated serving, which separates prompt processing and response generation across different GPUs for independent optimization. The framework integrates with backends such as vLLM, SGLang, and NVIDIA TensorRT-LLM, using etcd for service discovery and NATS for inter-component message passing. A deployment guide covers infrastructure setup, container configuration via Docker Compose, and two serving patterns using vLLM as the inference backend. The setup requires a minimum of one GPU for aggregated serving and four GPUs for disaggregated serving, along with the NVIDIA Container Toolkit and Docker Engine.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Deploy FastAPI with Gunicorn and Nginx on Ubuntu 24.04

A technical guide outlines the steps to deploy a FastAPI application on Ubuntu 24.04 using Gunicorn as the application server and Nginx as a reverse proxy. FastAPI, a modern Python web framework, supports asynchronous programming and ships with built-in interactive API documentation via Swagger UI. The setup uses Gunicorn with a Uvicorn worker class to handle ASGI support, while a systemd unit file manages the Gunicorn process as a persistent background service. Nginx sits in front of the application to handle incoming web traffic and route requests through a Unix socket. The deployment is completed by securing the application with a free SSL certificate, making it accessible over HTTPS on a custom domain.

0 comments Read more at DEV Community