AirLLM Enables 70B Model Inference on a Single 4GB GPU

·1 views

A new open-source tool called AirLLM has been released on GitHub, allowing large language models with up to 70 billion parameters to run inference on a single GPU with just 4GB of VRAM. This is a significant development as such models typically require far more GPU memory, often across multiple high-end cards. The project has gained early traction on Hacker News, attracting points and community discussion. By optimizing how model layers are loaded and processed, AirLLM makes powerful LLM inference accessible to users with consumer-grade hardware.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Solon Framework Auto-Tunes Thread Pools at Runtime Using CPU Core Count

Solon, a Java application framework, sets its HTTP server thread-pool properties to zero by default, meaning the system automatically calculates optimal values based on the host machine's actual CPU core count at runtime. The auto-tuner uses a single key flag, server.http.ioBound, to determine whether a workload is IO-bound or CPU-bound, then applies different scaling formulas accordingly. For IO-bound services, maxThreads is set to 32 times the core thread count, while CPU-bound services use a leaner multiplier of 8 to avoid unnecessary context-switching overhead. This approach eliminates the common problem of hard-coded or copy-pasted thread-pool values that may have been tuned for a different server environment. Developers can still override individual settings in app.yml, but the zero-config defaults are designed to remain suitable for most deployments without manual intervention.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

GitPython CVE-2026-67324: One-character shorthand bypasses remote code execution guard

A critical vulnerability in GitPython, tracked as CVE-2026-67324 and scored 9.8 on CVSS 3.1, allows attackers to bypass the library's built-in guard against dangerous git options. The flaw affects version 3.1.50, where the security check blocks the long-form flag --upload-pack but fails to catch its short-form equivalent -u, enabling arbitrary command execution even when allow_unsafe_options is set to False. Exploitation requires that user-influenced input reaches the multi_options parameter of Repo.clone_from, a scenario common in CI pipelines, web forms, and tools that accept repository URLs or build parameters. GitPython is downloaded roughly 254 million times monthly on PyPI, arriving as a dependency of widely used tools such as MLflow, DVC, and Semgrep, significantly broadening the attack surface. Version 3.1.51 patches the issue, and this is reportedly the third bypass of the same security barrier within a single year, all sharing the same root cause.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Alibaba Launches Qwen3.8-Max, a 2.4T-Parameter AI to Rival OpenAI and Anthropic

Alibaba on Monday unveiled Qwen3.8-Max, its largest artificial intelligence model to date, targeting enterprise use cases such as software engineering and complex reasoning. The model uses a mixture-of-experts architecture with 2.4 trillion total parameters, though only around 95 billion are activated per inference cycle, balancing performance with speed and cost efficiency. Open-weight versions are set to be released publicly via Alibaba's cloud studio platform the following week. Internal benchmarks shared by the company suggest Qwen3.8-Max performs competitively against leading models from OpenAI and Anthropic on coding evaluations such as SWE-bench Pro. Alibaba also claimed the model autonomously completed three coding projects without human input, with one project running continuously for 16 days from an empty folder to a finished software product.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Lets Anyone Build Apps, But Skipping Human Review Still Costs Dearly

A growing trend celebrates solo developers building apps over a weekend using AI tools, with no engineering team or formal background. However, critics argue that the absence of structured review processes — code reviews, QA testing, and managerial sign-off — leads to silent but consequential failures. Historical examples reinforce this: JPMorgan Chase's 2012 London Whale trading loss, partly traced to an unchecked spreadsheet formula error, cost the bank $6.2 billion. Similarly, a widely cited 2010 economic paper by Reinhart and Rogoff shaped years of government austerity policy before a graduate student discovered a formula error in 2013 that significantly altered its conclusions. The core argument is that review failures, not tool failures, are the recurring culprit — and AI-assisted development does not change that fundamental human blind-spot problem.

0 comments Read more at DEV Community