Janas software enables running large AI models by streaming from solid-state drives

·1 views

A new open-source software called Janas allows users to run large language models that exceed their computer's available RAM. It achieves this by streaming portions of the model, referred to as 'experts', directly from a solid-state drive (SSD). The software was created by developer Prabanta and is hosted on the GitHub platform. This method provides a practical workaround for hardware memory limitations when working with advanced AI models.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Open source local gate limits repair attempts to prevent free server overload

A developer proposes a local workflow to prevent excessive automated repair attempts on free-tier servers. The system uses two JSON configuration files to track change attempts and restrict resource usage. It caps repair attempts at a user-defined limit, typically three attempts for solo development work. The gate prevents processes from spawning unauthorized workers or binding to unapproved ports. This approach aims to maintain zero-cost hosting by avoiding unintended resource consumption.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Fiction Writing Scales Through Parallel Agents, Not Faster Models

The article argues that the main bottleneck in AI-assisted fiction writing is not model capability, but the serial, single-path approach typically used. It states that unlike a human writer, AI systems can work on many story elements simultaneously if the interface is designed for it. The key to scaling production, such as writing a novel in a week, is to increase the number of parallel writing 'lanes' and equally scale up the review capacity. This approach is said to be particularly well-suited for fan fiction, where the foundational world and elements are already established.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Security experts argue RBAC alone insufficient for AI agent permissions

Security professionals emphasize that traditional Role-Based Access Control is inadequate for AI agents due to their automated nature. Agents can amplify risks when granted broad permissions, as demonstrated by an incident where an agent drafted and sent a termination letter incorrectly. Experts recommend implementing four permission layers: identity verification, tool allowlists, data scopes, and action classifications. The proposed approach includes temporary, time-bound credentials instead of permanent API keys to minimize security risks. Organizations should conduct regular permission audits and implement regression testing for agent security controls.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Five methods to reduce costs for voice AI applications

The article outlines several technical strategies for lowering Text-to-Speech (TTS) API expenses. Developers can cache frequently used audio waveforms to avoid repeated synthesis costs. Batching multiple text prompts into single API requests reduces per-request overhead. Selecting lower-fidelity or low-latency voice models instead of premium neural voices can significantly decrease costs. Additionally, optimizing audio output by trimming silence and reducing bitrate further reduces storage and bandwidth usage.

0 comments Read more at DEV Community