Nightly LLM Evals on Short-Lived GPUs Using Runcrate's CLI

·1 views

A developer describes running LLM evaluations automatically every night instead of only before releases, so regressions surface early. The setup uses Runcrate's command-line tool to create a scheduled job that launches a fresh GPU, runs EleutherAI's lm-evaluation-harness, and shuts the machine down when the eval finishes. The example evaluates Qwen2.5-0.5B-Instruct on 200 ARC-Easy questions, with a run costing roughly a cent. The schedule includes a required maximum runtime to stop hung evals and a default setting that skips a run if the previous one is still active. The author reports running the commands on October 21, 2026, and notes that a PyTorch build must match the machine's CUDA driver version.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to turn a GitHub project into resume bullets recruiters find credible

A common resume mistake is listing a project's technologies without explaining the problem it solved or what the author personally did. The author recommends answering three questions: what problem the project addresses, which parts the candidate built, and what evidence a reviewer can verify. Using a hypothetical shift-swap app built for a café, the piece shows how a vague stack-focused bullet can become a specific, checkable one. It also argues that projects without real users can still be presented credibly by describing the engineering, tests, and documented decisions instead of invented metrics. The article closes with a checklist for reviewing project bullets and mentions a free resume-checking tool built by its author.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to test a data breach claim before believing or sharing it

Breach claims often spread on social media and forums within hours, frequently before anyone has verified them. Many are old leaks repackaged, scraped public data, or stitched-together lists, so a claim should be treated as an allegation until the organisation, CERT-In, a regulator or a credible researcher confirms it. Experts advise checking the source's track record and whether the record counts and fields are internally consistent. They also warn against downloading leaked samples, which carry legal risk and malware dangers. Individuals can check their own exposure through Have I Been Pwned and should change reused passwords and enable two-factor authentication. Under CERT-In's 2022 directions, specified cyber incidents must be reported within six hours of being noticed.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Small Businesses Can Move to the Cloud Step by Step Without a Risky Overhaul

A guide argues that moving a small business to the cloud should be broken into small, sequential projects rather than one large migration. It recommends starting with cloud email and file storage for remote access and automatic backups, then migrating business tools such as accounting, CRM, booking, and inventory one at a time while running old and new systems in parallel. The article stresses proper account setup, including individual logins, two-factor authentication, and role-based access, since misconfiguration causes most small-business cloud incidents. It also advises enabling automated backups and updates, testing a restore once, and reviewing cloud spending monthly to avoid rising costs. The piece is published alongside promotion of the author's cloud services firm, Cloudlift Tech.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Multi-Platform Publishing Pipeline Gains VS Project and Bluesky Support

A developer extended an automated content pipeline so a new project, VS, could publish its daily markdown to Medium, Substack, and Bluesky in a single CI run. The pipeline had skipped the VS batch because its metadata flags were still set to false, which left no posts on any platform and triggered a Slack alert. An initial hard-coded workaround in the publishing script worked once but undermined the pipeline's declarative design. The fix restructured content into dated folders with per-platform files and made platform detection data-driven through metadata flags. A new Bluesky flag defaults to false so payloads can be validated before posting.

0 comments Read more at DEV Community