SShortSingh.
Back to feed

Checking an ML benchmark for train/test duplicate rows with a hash-based audit

0
·3 views

A small time series dataset, PowerCons, has 180 training and 180 test rows, and 67 of the test rows appear in the training set unchanged. A 1-nearest-neighbour classifier scores perfectly on those duplicates, which inflates results without any visible error. The author proposes an efficient check: fingerprint each row by hashing its contents with the label excluded, then look up test fingerprints in a set built from the training rows. A small Python script that reads the archive's .ts files runs this check in a fraction of a second. Applied to the UCR and UEA archives, it finds leakage in 17 of 128 UCR datasets and 3 of 30 UEA datasets on their default splits.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Separating Liveness and Readiness Probes for Mock Servers to Prevent Flaky Tests

A mock server in a container should expose distinct liveness and readiness checks rather than one endpoint that reports success as soon as the process starts. Routes are only registered after the spec is read and examples are resolved, so a server can be reachable before it can actually answer the expected endpoints. Premature traffic in that window causes intermittent resets and 404s that look like test flakiness. Liveness should restart a dead process, while readiness should hold traffic back until the contract has loaded. Orchestration can wait on readiness, and a startup probe or generous failure threshold can accommodate slow loads without causing premature restarts.

0
ProgrammingDEV Community ·

Catalog Endpoints Do Not Establish a Checkout Contract for AI Agents

A product catalog can show an agent what items exist but grants no ability to purchase them, and this boundary is often overlooked in integrations. A QA run of one gateway found five store actions, covering product search and operations on existing orders, with no order-creation or checkout action discovered and no payment handlers advertised. The author argues this supports only a narrow conclusion about discovered capabilities, not whether purchases are possible or safe to add. Real checkout requires much more than product lookup, including cart construction, price freshness, payment authorization, confirmation, and idempotency. The author recommends that discovery separate read and mutation capabilities, enforce these boundaries server-side, and test both advertised and absent actions.

0
ProgrammingDEV Community ·

Arguseye adds Django template previews without starting an app server

Arguseye is adding a native Django workflow that lets developers preview a template with explicit template, engine and input data, without starting the application server. The feature is available on the feat/template-previews development branch, not in the published 0.0.15 npm and VS Code packages. Users define a template mapping and data profile in arguseye.templates.json, then run the CLI to capture a screenshot, accessibility output and errors. After editing, a diff command compares the result against the saved baseline and reports whether it is unchanged, changed or incomplete. The workflow renders templates with supplied context only and does not run views, middleware, context processors or database queries.

0
ProgrammingDEV Community ·

Low-bitrate videos can grow when re-encoded, so a quick ffprobe check flags them first

A developer built a small script that uses ffprobe to read header data and compute bits per pixel per frame, then labels each video SKIP or ENCODE before a batch re-encode. The threshold of 0.10 comes from an earlier test of 36 files, where all low-bpp files below 0.08 grew after re-encoding. On five public WebM clips from Wikimedia Commons, the script flagged only one file, a 1080p moose clip, which grew to nearly 2.8 times its original video size. The other four clips shrank by between 17% and 65% when re-encoded with x264 at CRF 23. The script runs entirely locally and only reads metadata, so it can screen a whole folder without modifying any files.