How GPQA Benchmarks Measure AI Scientific Reasoning Beyond Search and Recall

·5 views

Benchmarks like GPQA are designed to test genuine scientific reasoning in AI models by filtering out questions that skilled non-experts can answer using unrestricted web access and time. Domain experts write questions in their specialties, which are then vetted by peers and discarded if solvable through search, making the remaining score meaningful. The benchmark includes measured human baselines for both experts and non-experts, allowing a model's performance to be judged against a real human reference point rather than in isolation. A model scoring well above the non-expert baseline demonstrates real knowledge and multi-step reasoning across sciences, a capability absent in earlier model generations. However, the format has notable limits: multiple-choice structure allows elimination shortcuts, experimental design skills go untested, and contradictory or unverified real-world evidence is entirely absent from such question sets.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Cosine, Dot Product, or Euclidean? For Normalized Vectors, It Rarely Matters

A technical analysis on DEV Community explains that for most real-world vector search setups, the choice between cosine similarity, dot product, and Euclidean distance produces identical ranked results. When all stored vectors are normalized to unit length — as most modern embedding APIs return — cosine similarity and dot product yield the same numerical value, while Euclidean distance is a strictly monotone transform of cosine, meaning all three metrics return the same ordering. The practical distinction arises only when vector magnitudes vary: dot product favors longer vectors, which can skew results toward longer documents, while cosine ignores magnitude entirely. The key guidance is to use whichever metric the model card specifies, since models are trained against a particular scoring function and switching metrics can discard learned signals. A separate, common source of bugs is confusing similarity scores with distance scores — developers should verify whether a vector library returns similarity or distance before setting any threshold filter.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Google Gemini Can Stream iHeartRadio Podcasts via Natural-Language Requests

Google's Gemini AI assistant can now be used to stream podcasts on iHeartRadio through simple conversational prompts, such as asking it to play top interview podcasts on the platform. The feature allows users to discover and access content by describing what they want to hear, rather than searching for a specific show title. This approach combines content type, genre, and a named service into a single spoken or typed request, reducing the steps between user intent and media playback. Technical details such as regional availability, device support, account requirements, and data-sharing practices have not been publicly disclosed. The integration signals a broader shift toward AI assistants serving as the primary entry point for media discovery and consumption.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Python Asyncio Tarpit Traps 50,000 Malicious Connections Using Under 50MB RAM

A systems architect developed an asynchronous TCP tarpit in Python designed to exhaust attackers' resources rather than simply blocking them. The tool uses Python's asyncio library to accept malicious connections and respond with deliberate, phased delays — sending little to no data over extended periods — without consuming significant server resources. In benchmark tests simulating 50,000 concurrent malicious connections, the tarpit handled all of them in under 30 seconds while using approximately 45MB of RAM. The approach aims to paralyze automated scanning tools by keeping their sockets open and waiting indefinitely, raising the cost of an attack. The developer describes this tarpit as one component of a broader enterprise cybersecurity platform, codenamed TITAN, that is currently in development.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

TTFT vs Tokens Per Second: Why Optimising the Wrong Metric Wastes Effort

Two distinct metrics govern the latency of AI language model responses: Time to First Token (TTFT), which measures the delay before output begins, and Tokens Per Second (TPS), which measures generation speed once streaming starts. For interfaces where a human watches text arrive in real time, TTFT matters most, since readers absorb prose at roughly 5–6 tokens per second and most hosted models already exceed that threshold by a wide margin. Beyond that threshold, faster token generation offers no perceptible benefit to a reader, making every extra millisecond of TTFT the more meaningful bottleneck. In contrast, for agent pipelines or batch processes where no one watches intermediate output, TPS dominates total wait time and TTFT becomes negligible. Reporting a single blended latency figure obscures which lever actually needs pulling, often leading engineers to optimise the wrong variable and misread the results.

0 comments Read more at DEV Community