VIDRAFT Releases AX-RAY AI Safety Benchmark and Open Leaderboard on Hugging Face

·2 views

AI safety firm VIDRAFT has publicly launched AX-RAY, a diagnostic framework built around 117 evaluation items designed to identify latent risk behaviors in large language models and AI agents. The framework's central focus is detecting 'Causal Leakage,' a failure mode where a model's outputs are influenced by hidden or unintended causal cues rather than transparent reasoning. Two publicly available general-purpose AI models, including one from NVIDIA, showed anomalous signals on this dimension during evaluation. AX-RAY also features jurisdiction-aware assessments that map safety risks to national laws and cultural normative systems, with intended applications spanning finance, healthcare, robotics, and public services. The evaluation dataset is freely accessible on Hugging Face, and VIDRAFT clarifies that an anomalous signal indicates a potential vulnerability requiring further investigation, not confirmed harmful behavior.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Mistral Releases Shieldstral 1.0, a Self-Hostable 3B Moderation Model Under Apache 2.0

Mistral launched Shieldstral 1.0 on August 5, 2026, a 3-billion-parameter model built specifically to moderate text and image content before it reaches end users. The model runs on a single 16GB GPU in BF16 precision and is available on Hugging Face under an Apache 2.0 license, making it accessible to self-hosters without third-party API dependency. A key feature is its policy-adaptive design, allowing moderation rules to be defined in natural language within the prompt rather than requiring retraining when community guidelines change. Mistral reports strong benchmark performance, including 99.4% F1 on HarmBench and 97.7% on the multimodal VLGuard set, though these figures are vendor-reported and have not yet been independently verified. The model supports popular inference frameworks including vLLM, llama.cpp, and SGLang, and is intended as a specialized moderation layer alongside other models rather than a general-purpose replacement.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Isolation Forest vs GPT-4o: Choosing the Right AI Log Anomaly Detection Tool

Engineering teams moving beyond static threshold alerts for log anomaly detection face a key choice between statistical ML models like Isolation Forest and LLM-based tools like GPT-4o. Statistical models such as Isolation Forest and Prophet are fast, cost-effective, fully on-premise, and produce explainable results, but require manual feature engineering and regular retraining to stay accurate. LLM-based approaches like GPT-4o require no feature engineering and can interpret semantic context in logs, but raise serious data privacy concerns when logs contain API keys, PII, or session tokens. The practical decision hinges on three factors: daily log volume, whether real-time alerting or post-incident triage is needed, and whether sensitive data can legally be sent to a third-party API. Choosing the wrong approach for your constraints — especially around compliance — can create significant remediation challenges after the fact.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

15 Hours Lost: How Poor Docs and No Setup Scripts Hurt Junior Mobile Devs

A developer onboarded two junior engineers onto a React Native Expo project last week and logged every friction point they encountered. Both developers had web React experience but no mobile background, and each lost roughly 15 hours to avoidable setup and documentation failures. Common blockers included Node version mismatches, missing Xcode tools, misconfigured Android SDK paths, and confusion between Expo Go, dev builds, and EAS builds — none of which were addressed in the README. Undocumented auth patterns caused one developer to rebuild a token-refresh mechanism that was already running silently in middleware. The author concludes that a single setup script and a few focused documentation pages could have prevented nearly all of the lost time.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Options Buyer ML Model V2: How V1 Failed and What the Rebuild Fixed

A developer has documented the rebuild of a machine learning system designed to predict options-buying decisions in Indian equity markets, detailing why the first version (V1) consistently underperformed. V1 used a single XGBoost model to directly predict CE or PE direction from raw premium data, causing the model to learn noise as much as signal. Critical bugs — including a partition shift error that silently zeroed out a key feature and a rollup config that compressed training data from 295,000+ sequences to just 387 — contributed to a live paper-trading result of 31.6% win rate and a loss of ₹90,300. The rebuilt V2 separates the problem into narrow, task-specific model heads for underlying price movement and contract-level return targets, while applying strict hyperparameter tuning and an overfitting gate based on train-test metric gaps. The author emphasizes that the key lessons are to ask precise questions, keep decision trees shallow, and only promote models based on out-of-sample performance rather than training scores.

0 comments Read more at DEV Community