SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Running a 33B dense agent model on a single 24 GB GPU: offloading pitfalls and empty replies

0
·1 views

Nanbeige4.3-33B, an Apache-2.0 agent coding model released by the Nanbeige team at BOSS Zhipin on 24 October, was self-hosted on one RTX 4090 using llama.cpp and bartowski's GGUF files. Because the model is dense, offloading layers to CPU RAM sharply slowed generation, dropping a 4-bit run to 1.6 tokens per second, while a smaller 3-bit file fitting fully on the GPU ran at about 56 tokens per second. The 3-bit file still ran acceptably on a 16 GB RTX 2000 Ada, but the 2-bit version gave a wrong answer to a basic arithmetic check, so it was ruled out. With thinking mode enabled by default, some long replies came back with empty content and the full text in reasoning_content, while tool calls stayed reliable. The authors traced this to the chat template leaving a thinking block open, and found that disabling thinking mode placed the full answer in the correct field.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Caching pip downloads saves under a second in GitHub Actions; source builds cost far more

Adding cache: pip to actions/setup-python caches pip's download directory, which on a fast runner saves less than a second per install. The bigger time cost comes when a dependency lacks a wheel for the platform and must be compiled, which took 10.9 seconds in one measurement and is also stored in the cache. Tests on a latchkey-small runner showed the pip download cache cut installation time from 7.7 to 6.9 seconds. Poetry's cache option instead restores the whole virtualenv, skipping the install step as well as the download.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Hospital staff chatbot built from training materials and exam answer keys

A team built a bilingual English and Arabic chatbot to answer staff questions about the InterSystems TrakCare hospital information system. Their training corpus came from scraping Moodle assessment print-outs, which contained correct answers, yielding 433 verified question-answer pairs. They also transcribed 43 training videos with an open speech-to-text model and distilled them into topic files. The system uses a hand-built BM25 retrieval engine with source-trust weighting and an Arabic normalization and clinical synonym layer rather than a vector database. The chatbot is available as a live demo on a training environment without patient data, running on free open models.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

cdkd Pairs with AWS Emulator to Run CDK Application Code Locally Without Redeploying

A DEV Community article explains how combining AWS Emulator, a local tool that emulates AWS APIs, with cdkd, a faster alternative CDK deployment tool, streamlines CDK development. cdkd automatically wires up the connection to the emulator and resolves environment variables from CDK code, removing the need for a .env file. Developers can mix targets, using real AWS for some services while writing other data to the emulator. Application code, such as Lambda functions, runs directly from source, so changes take effect without redeploying. The author demonstrates the setup with a stack of a Lambda function, DynamoDB table, S3 bucket, and API Gateway, using AWS Emulator v1.0.322 and cdkd v0.349.0.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Docker COPY --from misspelled stage name is misreported as a registry pull error

A misspelled or renamed stage name in a Docker COPY --from instruction is not flagged as a stage error by BuildKit. Instead, BuildKit treats the name as an image reference and attempts a registry pull, producing an authorization-style failure. Numeric stage indexes behave differently and produce clearer errors, though they can silently point to the wrong stage after edits. Typical causes include renames that missed a caller, stages referenced from a different Dockerfile, and shifted indexes. The author recommends checking for a stage-name suggestion in the error message first and using named stages to avoid index drift.

0 comments Read more at DEV Community