SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

AI UI Testing Tools Score High in Demos but Struggle in Production, Builders Find

0
·1 views

The team behind AI UI testing platform TestStar ran a single 19-step real-world test case eight times and recorded wildly inconsistent results, with passes and failures stemming from entirely different causes each run. A deeper investigation revealed that while AI execution engines performed their tasks correctly, an underlying infrastructure issue — sync memory writes backing up a worker thread by step 19 — caused the platform itself to collapse. The builders argue that popular AI browser-testing tools like Browser-Use, Skyvern, and Midscene function as execution engines only, lacking what they call a 'Harness layer' covering failure handling, self-healing, observability, idempotency, memory, scheduling, and page knowledge. Real-world debugging also exposed that self-healing logic must verify intended user outcomes rather than trusting system-reported success, after a CSS bug caused the AI to click an 'Archive' button it believed was 'Delete'. The team warns that teams evaluating AI testing tools should ask about the Harness layer upfront, noting one organisation spent nine months picking a tool and then building missing infrastructure themselves.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How a Per-Project Token Quota System Stopped One Batch Job From Starving Others

Three projects sharing a single LLM gateway and a 10-million-token allowance ran into trouble on day nineteen when a batch summarization job consumed 7.1 million tokens in four hours, causing all other projects to fail with quota errors. The root cause was a shared token pool with no per-project attribution, meaning any single consumer could exhaust the entire allowance unchecked. To fix this, a per-project quota layer was built in Python, assigning each project its own daily token budget and per-minute request limit, with all limits summing to 9 million to preserve a 10 percent safety buffer. The system runs a pre-check before each upstream request using a conservative token estimate, rejecting over-budget requests with a 429 response, then reconciles actual usage from the API response afterward. This two-step approach ensures that a single bad estimate causes only a minor overshoot rather than a runaway consumption event.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds UrbanLease Appliance Rental Platform, Shares Key Technical Lessons

A developer built UrbanLease, a web platform allowing users to rent home appliances with features including rental history, payments, and separate user and vendor portals. Early in development, the project was migrated from SQLite to MySQL to better handle growing data complexity, requiring fixes to tables, columns, and database relationships. Feeling the initial version lacked differentiation, the developer added a rule-based smart recommendation system that filters and ranks appliances based on user inputs like budget and rental duration. A second feature, an appliance condition history log, was introduced to show prospective renters details such as past rentals, maintenance records, and service history. The developer noted that building a functional product is only part of the process, emphasizing that meaningful features should solve real problems rather than simply add more pages.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Study Shows Full-File AI Code Rewrites Consume 11x More Tokens Than Simple Generation

A developer burned through a 10-million-token free AI coding allowance in two days by repeatedly requesting full-file rewrites, prompting a structured investigation into token costs. Using MonkeyCode, an open-source AI coding platform offering 10 million free tokens and a free server, a reproducible benchmarking harness was built to measure real token consumption across common coding tasks. Results showed that a full-file rewrite costs roughly 3,300 tokens per run, compared to just 300 for a simple code generation task — an 11x difference. The benchmark covered five task types: code generation, debugging, test writing, refactoring, and full-file rewriting, with token estimates based on a four-characters-per-token heuristic. The findings highlight that deliberate task selection, rather than defaulting to rewrites, can stretch a 10-million-token allowance to cover tens of thousands of meaningful coding operations.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

AI-Written Tests Can Pass While Missing Critical Bugs, Mutation Testing Reveals

A developer merged an AI-generated 14-test suite for an ISO date parser without review, only to discover the tests failed to catch broken new timestamp formats added a week later. The tests repeatedly validated the same happy-path scenarios with minor input variations, leaving key edge cases completely uncovered. Experts warn that a fully green AI-generated test suite does not guarantee meaningful coverage, and that false confidence from such suites can be more harmful than having no tests at all. Mutation testing — which introduces deliberate code bugs to check whether tests detect them — offers a measurable way to assess suite quality, with a score below 50% indicating largely decorative tests. A workflow using the open-source MonkeyCode platform, which currently offers free server access and a 10-million-token allowance, is proposed to automate this audit on every agent-generated suite.

0 comments Read more at DEV Community
AI UI Testing Tools Score High in Demos but Struggle in Production, Builders Find · ShortSingh