Study Argues Asking LLMs for Confidence Scores Is Unreliable

·1 views

A blog post published on rumple.github.io argues that using large language models to self-report confidence scores is a poor practice. The piece was shared on Hacker News, where it gathered 48 points and 30 comments. The article's central claim is that the confidence values LLMs produce about their own answers do not reliably reflect how accurate those answers are. The source text provided is only metadata, so the specific reasoning and evidence behind this conclusion could not be verified in detail.

Read the full story at Hacker News

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Old repository notes steer AI agents as firmly as recent ones, study finds

A developer placed a 2019 note in a repository saying the team should not use TypeScript, then asked AI models to pick a stack for a new 2026 service. Every response avoided TypeScript, even though the note was seven years old and carried no stated conditions. Testing across models showed that a note's age had almost no effect on whether it was obeyed, and telling the model the note was old barely changed results. Hiding expired entries entirely led models to recommend TypeScript every time, indicating that expiry works best as a structural feature rather than a warning. The author built whynot, an MIT-licensed tool that stores rejected decisions as dated Markdown files in a repository, with expiry dates and revisit conditions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

CERT-FR Advisory Covers Palo Alto GlobalProtect macOS Privilege Escalation Flaw CVE-2026-0315

France's CERT-FR published advisory CERTFR-2026-AVI-1326 on 15 October 2026, relaying Palo Alto Networks bulletins issued the previous day, one of which covers CVE-2026-0315. The flaw is a local privilege escalation in the GlobalProtect app on macOS that lets a local user gain root and run commands with administrative rights. The vendor states no known workaround exists, while the record omits technical details of the defect and reports exploitation maturity as unreported. The author notes that the affected-platform claim originates with the vendor rather than the national CERT, and that a ZoomEye search returned 612 GlobalProtect hosts but zero CVE-indexed matches, figures that do not measure actual macOS exposure. The analysis concludes the advisory justifies a scheduled macOS patch but not an urgent incident narrative.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Guide to Extracting Public Company Emails and Social Links with Python

A DEV Community tutorial explains how to collect public contact emails and LinkedIn pages from a list of company domains without search engines or headless browsers. The method fetches each homepage and follows only same-site links that look like contact, about, or imprint pages, since German and Austrian sites legally require an imprint listing an email. It uses a regex-based parser that handles HTML entities, obfuscated addresses, and false matches like image filenames, and filters out social share buttons. The article also covers normalizing social profile URLs so duplicates collapse, and filtering for generic role-based inboxes such as info@ or sales@.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Healthcare AI Agents Should Be Designed to Hand Off to Specialists

Healthcare AI often spans fields with different terminology, evidence standards, and professional duties, so a system that can process multiple domains may still lack the authority to settle every issue alone. The article suggests developers define task boundaries, flag decisions needing specialist review, and set structured escalation paths. One example describes an agent spotting a possible medication problem, collecting lab results and medication history, and preparing an evidence summary for a pharmacist or clinician. Clear handovers matter as well, so the receiving professional knows the problem, the supporting evidence, open questions, and completed actions. The author argues that routing should account for clinical risk, task scope, and authorization, with consequential decisions kept within validated capabilities.

0 comments Read more at DEV Community