EvalPort Introduces 11-Grader System for Framework-Agnostic LLM Evaluation

·1 views

EvalPort, a new LLM evaluation platform, has released a grader system featuring 11 distinct types designed to cover the majority of real-world evaluation needs. Each grader type — ranging from exact_match and regex to llm_judge and json_schema — carries its own parameters, model references, and thresholds, making eval suites self-describing. The system was built to be framework-agnostic, meaning any evaluation framework can implement a subset of the grader types without compatibility issues. Test cases reference graders by ID, and multiple graders can assess the same test case, with scores recorded separately in a ResultSet. A custom grader type also serves as an escape hatch for use cases not covered by the 11 built-in options, and SDKs are available for both Python and JavaScript.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer cuts Chromatic visual testing costs tenfold by consolidating story snapshots

A software developer sharing on DEV Community reduced their visual testing bill by up to ten times by rethinking how Storybook stories are structured for snapshot tools like Chromatic and Percy. The core problem was that rendering every component variant, state, and theme as a separate story generated a large number of snapshots, since most per-snapshot tools bill by the number of stories rendered. The fix involved collapsing related variants — such as sizes, states, and themes for a single component — into one consolidated 'AllVariants' or 'AllStates' story that renders a grid of permutations in a single snapshot. The developer cautions that this approach requires balance: merging a single component's internal matrix works well, but independent pages or unrelated flows should remain as separate stories to avoid coupling unrelated visual diffs. The technique is described as tool-agnostic and applicable to any platform that charges or tracks on a per-snapshot basis.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Properly A/B Test AI Avatars Against Text Chat for Conversion Rates

A growing number of AI avatar platforms claim that voice and video avatars convert better than plain text chat, but rigorous independent testing of this claim remains rare. Unlike standard A/B tests, comparing avatars to text chat involves multiple simultaneous changes — interaction modality, response latency, and visual layout — making it harder to isolate the true variable. Experts recommend holding the underlying AI model and knowledge base constant across both variants, while tracking granular metrics such as engagement friction, session depth, and lead quality rather than aggregate conversion rate alone. A notable pattern observed is that avatar variants can drive longer sessions without necessarily improving completed-lead rates, a nuance that overall conversion figures would obscure. Vendor case studies typically benchmark avatars against having no chat widget at all, not against equivalent text chatbots, making self-run controlled tests essential before committing to a premium-priced avatar solution.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

TryHackMe CTF Exposes Azure Cloud Privilege Escalation via Hardcoded SAS Token

A TryHackMe capture-the-flag challenge called CryptoCabana simulates a real-world Azure cloud compromise stemming from poor security practices. Participants begin by discovering a hardcoded SAS token with read and list permissions embedded in client-side JavaScript on a static Azure website, granting unauthorized access to the storage account. Enumerating the storage containers reveals a hidden vault container holding a service principal credential file, which is then used to authenticate directly to Azure. From there, contestants explore an Azure Key Vault containing four secrets, including three key shards and a protected master key, with the flag reconstructed by retrieving older versions of a rotated shard. The challenge highlights critical cloud security risks such as long-lived tokens, publicly exposed credentials, and insufficient secret rotation practices.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Complete Guide to Commonly Used Expressions in n8n Workflows

n8n supports JavaScript-like expressions wrapped in double curly braces that allow users to dynamically reference and transform data within automation workflows. These expressions cover a wide range of use cases, including accessing JSON fields, referencing outputs from specific nodes, and handling binary data. Users can also work with date and time values using built-in variables like $now and $today, along with Luxon-based formatting methods. Conditional logic is supported through expressions like $if, $ifEmpty, and the ternary operator, while string and array manipulation is possible via standard JavaScript methods. The reference guide serves as a practical cheat sheet for both new and experienced n8n users building complex workflow automations.

0 comments Read more at DEV Community