Small RAG reranker experiment shows a tie, offering a verified pipeline but no proven gain

·16 views

A developer fine-tuned a pretrained multilingual reranker and checked that its weights changed and that the saved checkpoint reloaded correctly. On a held-out synthetic slice of three questions, only two answerable, the adjusted and original rerankers both scored an nDCG@5 of 1.0, so the measured ranking gain was zero. The experiment isolated reranking by keeping the embedding model and the Qwen answer generator fixed, comparing four variants with and without retrieval and reranking. The work ran locally on CPU and is still in progress, with no real-domain fine-tuning, sealed real-data comparison, or human review of answers yet completed. The author concludes that the evidence supports a working experimental pipeline rather than any claim of improved answers.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer open-sources four MIT-licensed tools for AI-driven e-commerce work

Naser Sadrmanesh, an AI product engineer and store founder, has released four small open-source projects, all under the MIT license. The tools were built from utilities he repeatedly needed for client online stores. One is llm-search-eval, which compares search system changes using a shared label pool, LLM judges, and statistical tests. Another is woocommerce-mcp, which gives AI agents access to WooCommerce stores with read-only defaults, preview-and-confirm writes, and audit logging. A third, commerce-agent-skills, is a library of 29 Markdown skills that guide coding agents toward correct e-commerce implementation details.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Debugging a failing Docker container by committing it and overriding the entrypoint with a shell

A developer using Docker Compose found that a Django container kept exiting shortly after launch, showing exit code 127 in the docker ps output. To investigate, they committed the failed container to a new image so its state could be preserved. They then started that image with the entrypoint overridden to launch an interactive shell, using /bin/bash or /bin/sh if bash is unavailable. This gives access to run diagnostic commands inside the container's environment, with the specific checks depending on the problem.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Planning Active Directory Protected Users membership and managing its exceptions

Microsoft's Protected Users group applies fixed protections against credential theft, such as blocking credential caching, NTLM and Digest delegation, and limiting Kerberos to newer encryption. Because these restrictions cannot be toggled per account, adding a user is a compatibility decision as well as a security one. The article recommends first inventorying authentication dependencies through logs, then running a small pilot that tests interactive sign-in, administration, Kerberos tickets and delegation. Organizations should keep a tested break-glass account, record approvals and rollback steps, and review membership regularly so the group stays small and current. Protection only applies on supported hosts and domain controllers, so the platform should be verified before reporting an account as protected.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Treat AI Agent Tools Like Public APIs to Prevent Duplicate Actions

A developer's first production AI agent issued the same refund three times after a payment API returned a 502 error and the model retried the call. The root cause was a missing idempotency key, a bug familiar from standard HTTP client code rather than a language model problem. The article argues that the model should be treated as an unpredictable API caller, so tools need the same safeguards as public APIs, including versioning, timeouts, idempotency, scoped credentials, and rate limits. It recommends classifying every tool as read, write, or destructive, with that classification enforced in code rather than documentation. The author also advises generating idempotency keys in the application, since models will retry calls for many reasons.

0 comments Read more at DEV Community