LLM Trading Bots Beat Benchmark, But Pre-Set Rules Declare Result Inconclusive

·1 views

A developer ran a two-month paper-trading experiment using LLM agents managing a simulated $100 cryptocurrency portfolio across two seasons, with no real money ever at stake. In Season 2, all four AI agents — using Claude Opus and GPT via Codex in autonomous and structured configurations — outperformed a passive 50/50 WETH/USDC benchmark by 3 to 10 percentage points at the 30-day mark. However, pre-committed decision rules required consistent outperformance across two 30-day windows with a minimum 2 percentage point net edge, a threshold the agents failed to meet within the frozen evaluation window. Inside that designated window, both primary agents posted negative returns and underperformed cash and the passive portfolio, triggering a formal 'STOP' verdict under the pre-registered rules. The author argues the 'inconclusive' outcome is the most valuable finding, as it demonstrates how pre-registration prevents selectively interpreting favorable data as proof of genuine trading edge.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Agent Loop Bug Caused Duplicate Webhook Writes Due to Truncated Tool Output

A developer debugging a webhook worker discovered that an AI agent loop was triggering duplicate database inserts after receiving truncated tool output — a cut-off JSON response that lacked a closing brace. The agent, unable to confirm the first write had succeeded, reissued the mutating call, resulting in two records for the same delivery ID. Initial investigation wrongly blamed at-least-once message delivery, but raw payload logging revealed the real culprit was an incomplete receipt from the tool itself. The fix involved adding a strict parse-and-validate gate before any insert, requiring a fully formed JSON object with an explicit acknowledgment field before allowing the mutator to run. The experiment was conducted on a free remote server to avoid billing costs during overnight testing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Study: GitHub Copilot Boosts Code Functionality But Not Security, Developers Warned

A practical security review guide for teams building customer-support copilots using OpenAI API and Next.js has been published, drawing on findings from an empirical study of 44 developers. The study found that GitHub Copilot improved functional correctness and slightly reduced some insecure coding patterns, but did not significantly improve secure API usage. Researchers also noted that developers rarely raised security concerns while working with AI assistance. The guide emphasizes that a working prototype is not automatically a secure product, and that AI-generated code should be treated as candidate work requiring independent human review. Teams are advised to define explicit security requirements, conduct structured code reviews, and validate against official vendor documentation before release.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Audit of 30 Dev Community Posts Surfaces Unexplained File With Unknown Data

A structured AI-assisted review of 30 records on DEV Community, involving six contributors identified as Mark, Derek, Lena, Leo, Alex, and P, concluded with an anomalous file appearing in the directory that no author claims to have created. The file contained numerical data — including counts for posts, reactions, comments, views, and followers — alongside 15 names, none of which match any of the six contributors in the reviewed records. Analysts noted two names containing 'Alex' and one name differing from contributor 'Leo' by a single character, but could not determine whether these represent the same individuals or coincidental overlaps. Every label in the file could be read, but its referents remain unclear — views of what, followers of whom — and the sponsorship count was zero. The file carried no narrative, no signature, and no verifiable origin, leaving its source and purpose undetermined.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Selenium's Parallel Chrome Instances Can Trigger Windows Account Lockouts

A developer discovered that running multiple Selenium-controlled Chrome instances in parallel was causing repeated Windows account lockouts, despite the correct password always being used. Chrome contains Windows-specific logic that calls the LogonUser API with an empty password to check authentication state, generating failed login attempts recorded as Event ID 4625 in the Windows Security log. When many ChromeDriver instances are launched simultaneously, each creating a fresh temporary profile, the volume of failed attempts can quickly exceed the Windows account lockout threshold. The root cause was identified by examining the Security event log, which consistently pointed to chrome.exe as the source process. The recommended fix is to assign each Selenium worker a dedicated persistent Chrome profile, which both prevents repeated failed authentication attempts and avoids redundant first-run initialization overhead.

0 comments Read more at DEV Community