SShortSingh.
Back to feed

Why checking for empty strings after Unicode normalization can silently drop payee data

0
·1 views

A developer building a bank-file conversion tool discovered a subtle bug where Unicode normalization (NFKD) converted fullwidth Chinese punctuation into standard ASCII punctuation, leaving a non-empty string even after all meaningful characters were lost. Because the code checked for an empty string to flag fully lost payee names, transactions with names like '北京,上海' slipped through as a false 'partial loss' warning instead of a 'complete loss' alert. This meant QBO files were imported with correct dates and amounts but blank payee fields, with no accurate warning shown to users. The fix replaced the empty-string check with a regex test for at least one alphanumeric character, correctly identifying strings containing only leftover punctuation as fully emptied. The case illustrates a broader pitfall: wherever Unicode normalization precedes an emptiness check, the real question should be whether any meaningful content survives, not merely whether any characters remain.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Mercury 2.5 Tops LLM Speed Charts at 1,107 Tokens Per Second in September 2026

Inception Labs' Mercury 2.5 has emerged as the fastest large language model accessible via API as of September 2026, delivering 1,107 tokens per second by vendor report and 440 tokens per second at median latency per OpenRouter's third-party telemetry. The model uses a diffusion-based architecture that processes token blocks in parallel rather than sequentially, giving it a structural speed advantage over autoregressive rivals like GPT-5.6 Luna and Claude Haiku 4.5. Mercury 2.5 is priced at $0.20 per million input tokens and $0.75 per million output tokens, undercutting competitors on output cost, though an 80% launch discount expired on 8 September 2026. GPT-5.6 Luna remains the preferred option for general-purpose chat due to its larger context window and broader ecosystem maturity, despite its slower speed of 129 tokens per second. Claude Haiku 4.5 and Gemini 3.5 Flash-Lite occupy the middle ground, with Flash-Lite offering competitive speed at a higher output price and Haiku favoured for Anthropic's instruction-following behaviour rather than throughput.

0
ProgrammingDEV Community ·

GLM-5.3-Flash Leads Agentic Coding as Top Open-Source Flash LLMs Split by Use Case

A September 2026 comparison of leading open-source flash-tier language models finds each excels in a distinct deployment scenario. GLM-5.3-Flash, a 320B mixture-of-experts model released by Z.ai on 26 August 2026, tops agentic coding benchmarks with a Terminal-Bench 2.1 score of 84.3 and DeepSWE v1.1 score of 63.4, attributed to its ability to maintain task state across multiple tool calls. DeepSeek V4 Flash, a 284B MoE model refreshed on 31 July 2026, is the most cost-efficient option for high-volume workloads under an MIT licence. MiniCPM5-2B, released on 7 September 2026, wins the on-device category with just 2.52 billion parameters and support for llama.cpp and Ollama. Qwen3.8-Flash-Next from Alibaba ranks close behind GLM on general intelligence and offers the longest native context window at 262K tokens, though analysts caution that several headline benchmark figures are vendor-reported and have scored lower in independent evaluations.

0
ProgrammingDEV Community ·

OpenBMB Releases VoxCPM2, a Free Self-Hosted AI Voice Model Supporting 30 Languages

OpenBMB launched VoxCPM2 in April 2026, a 2-billion-parameter, Apache-2.0 licensed text-to-speech model that runs entirely on a user's own hardware. The model supports voice cloning from short audio clips, text-prompted voice design, and standard speech synthesis across 30 languages, outputting audio at 48 kHz. Unlike token-based systems, VoxCPM2 uses a tokenizer-free diffusion-autoregressive architecture, which its developers say better preserves natural speech qualities like pacing and emotion. It integrates with an OpenAI-compatible API endpoint, meaning developers can switch from hosted services such as ElevenLabs largely by changing a base URL. While ElevenLabs is noted for greater consistency and polish, VoxCPM2 eliminates per-character fees and keeps user data off third-party networks for routine generation workloads.

0
ProgrammingDEV Community ·

GuardianKane Dashboard Makes AI Agent Testing Gaps Visible in Real Time

GuardianKane, originally a terminal-based code verification hook, has been rebuilt with a visual dashboard running at localhost:4173 to give developers deeper insight into what an AI agent has touched during development. The dashboard features four tabs — Code graph, Memory graph, PRD graph, and Kane activity — along with a chat panel, all drawing data from small JSON files written by hooks into a hidden project directory. A key metric surfaced by the PRD graph highlights the gap between designed tests and proven ones: while 100% of acceptance criteria have a test designed for them, only 53% have actually passed in a browser. Touch edges, recorded silently since the original hackathon version, power the code graph by mapping exactly which files an agent modified per task. A minor bug was also discovered during the writeup — GuardianKane's own installed files appear in the project graph by default, a flaw the team plans to fix by enabling automatic exclusion.