Study finds AI models contradict themselves 74% of the time on software recommendations

·1 views

A developer built an automated testing harness that asks eight AI models the same B2B software buying question every month across sixteen product categories. Across all runs, the eight models never unanimously agreed on a single tool in any category, revealing deep inconsistency in AI-generated recommendations. More strikingly, individual models changed their own top pick approximately 74% of the time when asked the same question in a new session with no other variables altered. The researcher has published all raw data, prompts, and logs under an open CC-BY license and reruns the experiment monthly to track how model preferences shift over time. The findings carry a direct warning for marketers pursuing AI search optimization: a brand ranking first in one ChatGPT session may not rank first in the next, making single-snapshot strategies unreliable.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

React Compiler in React 19 Auto-Memoizes Code, Reducing Manual Hook Overhead

React Compiler, formerly known as React Forget, shipped as a stable feature with React 19 and automatically handles memoization of components, hooks, and callbacks by analyzing data flow at build time. Developers no longer need to manually write dependency arrays, as the compiler infers them and inserts the equivalent of useMemo, useCallback, and React.memo automatically. The compiler is deliberately conservative — if it cannot safely prove that memoization is correct, it leaves the component unchanged rather than risk introducing bugs. An escape hatch directive called 'use no memo' allows developers to exclude specific functions from compiler optimization when needed. Before enabling the feature, developers are advised to run npx react-compiler-healthcheck@latest to identify violations and assess how much of their codebase can be optimized.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Backend Engineer Cuts AI Costs 40x by Switching from OpenAI to Cheaper Alternatives

A backend engineer reduced their team's monthly LLM spending from roughly $500 to around $12.50 by migrating away from OpenAI's GPT-4o to DeepSeek V4 Flash, which charges $0.25 per million output tokens compared to OpenAI's $10.00. The engineer evaluated several alternative models including Qwen3-32B and DeepSeek V4 Pro, finding price differences as large as 40 times cheaper than the GPT-4o baseline. Blind user tests reportedly showed 80% of users could not distinguish between GPT-4o and the cheaper alternatives at their workload scale of roughly 40 million input and 10 million output tokens per month. The technical migration took only about three hours because Global API uses an OpenAI-compatible endpoint, requiring just two configuration changes — the API key and base URL. The engineer shared the migration journal across three production services to guide others through a similar cost-cutting transition.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Reciprocal Rank Fusion explained: how hybrid search merges retriever results

Hybrid search systems use multiple retrievers — such as keyword-based BM25 and dense embedding models — each returning their own ranked list of documents for a query. Merging these lists by averaging raw scores fails because retrievers operate on vastly different numerical scales, causing the one with larger numbers to dominate the result. Reciprocal Rank Fusion (RRF) solves this by discarding raw scores entirely and instead converting each document's rank position into a value of 1/(60 + rank), then summing these values across all retrievers. The constant 60 dampens the gap between top-ranked and lower-ranked documents, ensuring a document ranked highly by multiple retrievers rises to the top of the final merged list. However, RRF has tradeoffs and is not always the right tool, particularly in scenarios where the magnitude of scores carries meaningful signal that rank position alone cannot capture.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Context Engineering, Not Better Models, Is the Key to Reliable AI Agents

A growing body of practice in AI development points to 'context engineering' as the critical discipline determining whether AI agents succeed or fail, with most failures traced to poor information pipelines rather than model limitations. The field splits into two distinct challenges: personal context, which tailors AI to an individual's preferences and history, and shared context, which governs how organizational knowledge is safely accessed across large teams. Personal context systems rely on tools like style files, memory services, and knowledge bases, while enterprise shared context demands stricter controls around permissions, data provenance, and accuracy. A key technical constraint shaping both approaches is the finite 'context window' each model call operates within, where overloading it with loosely relevant data degrades performance — a phenomenon called context rot. Effective context engineering is therefore less about feeding models more information and more about curating the right information at the right moment.

0 comments Read more at DEV Community