SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

DEV Community lab proposes measuring routing tasks between free model endpoint and self-hosted server

0
·1 views

A DEV Community article presents a 90-minute workshop for teams to decide which execution path handles each class of task, based on their own measurements rather than public leaderboard scores. The lab uses a frozen prompt set, a JSONL evidence log, and a twelve-row ledger covering three task kinds across four execution paths. Only schema-valid responses count as successes, so fast malformed outputs are treated as failures. The workshop culminates in a routing policy table that participants can rerun after endpoint changes. A reference harness script is included, though it has not been run against a live endpoint and must be adapted to the endpoint in use.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Fault-injection drill tests offline fallback in local-first apps by cutting the network mid-request

The article argues that a local-first app only proves its offline behavior when the network fails during an in-flight request, not between requests. It recommends a stdlib-only Python HTTP stub that can simulate four failure modes: connection reset, stalled body, truncated response, and normal success. Running the client against each mode is meant to expose bugs such as a truncated response being parsed and reported as success. The author stresses writing explicit assertions before testing, including a local-write latency budget, a check that queued items are not lost, and a check against duplicate sends on retry. Generating an idempotency key before the first attempt is presented as the safeguard against double-writes in reset scenarios.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Tabs in One: un manager Chrome gratuit care regrupează filele pe o singură coloană și eliberează RAM

Dezvoltatorul Tabs in One, o extensie gratuită pentru Chrome, a lansat un instrument care mută toate filele deschise pe o singură pagină cu o coloană lizibilă. Motivația este că bara orizontală de file devine greu de folosit când sunt deschise peste zece file, iar filele consumă multă memorie. Închiderea filelor originale după mutare eliberează imediat RAM-ul, iar utilizatorii pot restaura oricare sau toate filele cu un singur clic. Extensia oferă backup-uri automate, export și import în format JSON sau HTML și nu necesită cont sau abonament. Ea nu sincronizează sesiunile între dispozitive, iar codul sursă este deschis pentru verificare.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer builds SHADOW, a local platform for supervised AI agents on Windows

A computer science junior describes building SHADOW, a platform for deploying AI agents that run on the user's own PC with their own API keys. Each agent has its own chat, model, permissions, and schedule, and works from a goal by first producing a plan the user must approve. Agents can only access folders and run commands the user explicitly grants, must show risky commands before executing, and cannot control the screen without the user starting it. The author says the design grew out of frustration with manually coordinating separate chat, documentation, and terminal tools. SHADOW is free on the Microsoft Store for Windows 11 and requires no account.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Keeping a human approval gate between LLM agent proposals and system writes

An AI agent can prepare a consequential change without being permitted to carry it out, which lets organizations automate work while keeping a person accountable. The article uses a supplier onboarding example, where the agent validates data and drafts an ERP record but cannot create the supplier or release payments. It recommends a state machine with a persisted, versioned proposal that reviewers approve exactly, along with separate execution endpoints that accept only valid approval tokens. Approval records should capture identity, payload hash, decision and expiry, and executors need idempotency keys and reconciliation. The author suggests testing rejection, expiry, modification, duplicate and outage paths, and piloting one low-volume workflow with measured outcomes.

0 comments Read more at DEV Community
DEV Community lab proposes measuring routing tasks between free model endpoint and self-hosted server · ShortSingh