How to Install Rust Toolchain for vLLM on AWS Graviton2 G5g Instances

·2 views

A technical walkthrough details how to install and configure the Rust toolchain for running vLLM on AWS G5g instances, which pair a Graviton2 (aarch64) processor with an NVIDIA T4G GPU. Since pull request #40848 was merged, vLLM includes a 14-crate Rust workspace that builds two key artifacts: the axum-based vllm-rs HTTP server binary and a PyO3 Python extension module. The Rust toolchain is a hard build-time dependency because vLLM's setup.py imports setuptools_rust at module scope with no opt-out, meaning metadata generation fails without it. The guide was tested on EC2 g5g.xlarge and g5g.4xlarge instances in us-east-1a, using vLLM 0.27.2rc1 and rustc 1.97.1. The resulting vllm-rs binary acts as a drop-in replacement for vLLM's Python FastAPI server, running as its own 50 MB aarch64 process.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Google cuts Gemini 2.5 Flash token costs by 50%, boosts coding and automation

Google has released Gemini 2.5 Flash, a developer-focused AI model with improved performance in coding, automation, and multi-step workflows. The new model is priced at $0.75 per million input tokens and $3.75 per million output tokens, roughly half the cost of its predecessor. The release comes just three weeks after the previous version, reflecting Google's strategy of rapid iteration for its Flash series. Benchmark results show notable gains in debugging, code generation, and autonomous agent tasks compared to version 3.6. Google says the price reduction is intended to make large-scale production deployments more accessible for businesses of all sizes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Referral Traffic Hits 1.08% of Web Visits but Understates Broader Influence

A Conductor report analyzing 13,770 domains across 10 industries found that AI-driven referral traffic accounted for 1.08% of total website visits between May and September 2025. The channel grew at roughly 1% month over month during the study period, signaling steady but modest expansion. ChatGPT dominated AI referrals, generating approximately 87.4% of that traffic, making overall figures heavily dependent on a single platform. Experts caution that referral data alone understates AI's true impact, as users may encounter brands in AI-generated answers without ever clicking through to a website. Industry context also matters significantly, with IT sectors seeing AI referral shares as high as 2.8%, well above the cross-industry average.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Qwen 3.8 27B Local Setup: GGUF Sizes, KV Cache Savings, and a Critical Flag

Alibaba released Qwen 3.8 27B under an Apache 2.0 licence on 13 August, making it freely usable for local deployment, while a separate 2.4-trillion-parameter MoE variant launched under a more restrictive licence. The model uses 64 layers but reserves a KV cache for only 16 full-attention layers, cutting per-token cache cost to 64 KB versus 256 KB on a conventional dense model, making long contexts far more memory-efficient. Quantised GGUF versions range from a 9 GB 2-bit build to a 53.8 GB BF16 file, with the Q4_K_M quant at around 17 GB considered the practical sweet spot for 24 GB GPUs. The model also natively handles images and video without additional wrappers, and ships a separate 0.9 GB vision encoder file. Users running llama.cpp must include the --jinja flag to load the model's custom chat template, as omitting it causes erratic output that is frequently mistaken for a corrupted download.

0 comments Read more at DEV Community

ProgrammingHacker News ·

Study finds coffee consumption linked to metabolic health and sex hormone levels

A new study from the University of Oulu has found connections between coffee consumption and both metabolic health and sex hormone levels. The research suggests that drinking coffee may have measurable effects on hormonal and metabolic markers in the body. The findings add to a growing body of evidence exploring how dietary habits, particularly coffee intake, influence physiological processes. Details of the study were published by the University of Oulu, a Finnish research institution known for health and population studies.

0 comments Read more at Hacker News