GuardianKane Dashboard Makes AI Agent Testing Gaps Visible in Real Time

·5 views

GuardianKane, originally a terminal-based code verification hook, has been rebuilt with a visual dashboard running at localhost:4173 to give developers deeper insight into what an AI agent has touched during development. The dashboard features four tabs — Code graph, Memory graph, PRD graph, and Kane activity — along with a chat panel, all drawing data from small JSON files written by hooks into a hidden project directory. A key metric surfaced by the PRD graph highlights the gap between designed tests and proven ones: while 100% of acceptance criteria have a test designed for them, only 53% have actually passed in a browser. Touch edges, recorded silently since the original hackathon version, power the code graph by mapping exactly which files an agent modified per task. A minor bug was also discovered during the writeup — GuardianKane's own installed files appear in the project graph by default, a flaw the team plans to fix by enabling automatic exclusion.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds Magic: The Gathering rules agent and tests it against keyword search

A developer created JudgeStack, an AI agent that answers Magic: The Gathering rules questions by identifying the correct authoritative source for each query type, such as Oracle card text, Comprehensive Rules, or dated format announcements. The project was submitted to the DEV x Sanity Challenge and uses two Sanity Context MCP endpoints to separate structured document queries from the full Comprehensive Rules file. To evaluate its accuracy, the developer ran a blind comparison between basic keyword retrieval and the structured Sanity Context approach across ten previously unseen questions. The structured system answered 9 out of 10 questions correctly, while keyword retrieval managed only 1 out of 10, with 7 answers relying on evidence that was never actually retrieved. The corpus underlying JudgeStack spans 496 documents covering cards, printings, legality claims, rulings, and wording differences, with IP considerations limiting republication of the full rules text.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI-Reliant Developer Attempts to Write Python Code Alone, Confronts Basic Gaps

A developer who has shipped over 26 projects by directing AI agents decided to write Python code entirely by hand for the first time while applying to Stanford's free Code in Place X introductory programming course. Despite prior exposure to a DataCamp Python course, the author could not independently complete five basic programming tasks, confirming eligibility as a beginner applicant. Working through the application's coding lessons with Claude strictly as a tutor — typing every line personally — revealed repeated errors including misplaced symbols, missing spaces, and a single misspelled letter in a prompt that caused all three automated tests to fail. A trivia program the author wrote independently returned incorrect output, confidently naming Kareem Abdul-Jabbar as the Lakers' all-time leading scorer instead of Kobe Bryant. The experience highlighted a distinction between directing AI agents to produce working tools and developing the foundational understanding that comes from writing and debugging code oneself.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Better Workflow for Evaluating Technical Sources Before Trusting Them

Developers spend significant time reading third-party technical content such as documentation, Stack Overflow answers, and tutorials, but not all of it applies to their specific environment. A key gap exists between finding an answer online and finding one that matches the current runtime, OS, and package versions in use. The article recommends documenting your environment first — including runtime versions, dependencies, and exact error messages — before searching, to narrow results and assess source relevance. It also advises categorizing error messages by type (permission, missing file, timeout) and applying fixes one at a time to isolate the true cause. Older resources can still be useful for concepts, but commands and configurations should always be verified against the latest official documentation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Dev Team Ditches GitHub Cloud Minutes With Self-Hosted VirtualBox CI Runner

A development team split their monorepo into five separate repositories, only to find that independent CI pipelines rapidly consumed their free GitHub Actions minutes allowance. Rather than pay for additional cloud minutes, they set up a self-hosted runner on a standard Windows office laptop running Ubuntu inside a VirtualBox virtual machine. The single VM was configured to serve all five private repositories in turn, with each job preceded by an automatic rollback to a clean snapshot to prevent build contamination. GitHub's built-in ephemeral runner flag was insufficient on its own, as it does not restore the filesystem, prompting the team to write a custom REST API polling script as an external controller. Early stability issues, including zombie VirtualBox processes and a watchdog script that inadvertently blocked its own health checks, required additional fixes and led to a multi-day debugging effort the team plans to detail in a follow-up post.

0 comments Read more at DEV Community