SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

POCKET GGUF 모델의 실제 램 사용량을 비트 수와 KV 캐시로 분해한 분석

0
·2 views

로컬에서 모델을 실행할 때 파일 크기만으로 램을 정하면 문맥 길이와 구조에 따라 예상과 어긋날 수 있다고 설명한다. 저자들은 POCKET GGUF 파일 8개의 파라미터 수와 파일 크기를 이용해 가중치당 평균 비트를 계산했으며, 같은 Q2_K 이름이라도 모델에 따라 3.53비트와 2.99비트로 다르게 나타났다. KV 캐시는 문맥을 쌓는 층이 일부에 그쳐 262K 토큰에서도 5~6GiB 수준에 머문다고 밝혔다. 180B 모델은 파일의 대부분이 전문가 가중치와 임베딩 조회표이며, 토큰당 실제로 읽는 가중치는 파일의 5.7%에 불과하다고 전했다. 이를 바탕으로 문맥 길이별 램 예산표를 제시하고, 여유 메모리를 감안해 빌드를 고르는 방법을 안내했다.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Tailscale vs Headscale: Trade-offs of Self-Hosting a Mesh VPN Control Plane

A DEV Community guide compares the official Tailscale service with Headscale, an open-source, self-hosted replacement for its coordination server. It explains that Headscale only substitutes the control plane, so users still run the official Tailscale clients while peer-to-peer WireGuard tunnels remain unchanged. The article covers deployment needs such as a domain name, valid TLS certificates, a static IP or dynamic DNS, and a reverse proxy, and provides a sample Docker Compose setup. It also outlines how to point Linux and iOS clients at a custom login server and notes that Headscale has no built-in web interface, so administration is done via the command line. The author recommends the free Tailscale tier for most users, reserving Headscale for those who need full control over their network infrastructure.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Synthetic identity fraud targets AI-driven customer onboarding at lenders and fintechs

Criminal rings are exploiting automated digital verification pipelines at credit unions, lenders, and fintechs by blending real Social Security numbers with AI-generated licenses, deepfake faces, and cultivated credit histories. Legacy KYC rules often fail to detect these hybrid fabrications, which do not trigger the immediate alerts that conventional identity theft does. Fraudsters build fabricated credit profiles over 12 to 24 months before draining multiple lending institutions at once, typically leaving write-offs and vanished perpetrators behind. Experts recommend stronger controls, including inspection of document artifacts, hardware-anchored biometric liveness checks, and multi-bureau credit cross-referencing. The article describes these attacks as an industrial-scale operation that uses commercial generative AI tools and camera emulation to bypass existing safeguards.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Exposed wg-easy admin panels pose outsized VPN risk despite small public footprint

Researchers found only three ZoomEye title matches for wg-easy, a small web interface for managing WireGuard, but warn that a low count does not mean low risk. The admin UI can create peers, manage keys, and retrieve VPN configuration, so a compromised session can grant network access without touching host firewalls. Documentation for a pre-release version reportedly assigns admin permissions to externally authenticated users, so operators should verify role behavior in the exact version they run. Recommended defenses include keeping the admin UI behind an independent authenticated access layer with MFA, restricting source networks, and isolating the Docker socket from the web container. Operators should also treat peer private keys as credentials, audit peers after any suspected admin compromise, and monitor logins, peer creation, and configuration downloads.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Globo Nordeste viewers see Mato Grosso morning show during December 2015 signal outage

On December 29, 2015, a signal failure at Globo Nordeste led to an unexpected broadcast swap. Instead of the Pernambuco-based Bom Dia morning program, viewers were shown Bom Dia Mato Grosso, aired by TVCA. The change happened because most of the network's equipment had been damaged. The incident was reported after it circulated as a tweet that captured the mix-up.

0 comments Read more at DEV Community