SShortSingh.
Back to feed

Why checking for empty strings after Unicode normalization can silently drop payee data

0
·15 views

A developer building a bank-file conversion tool discovered a subtle bug where Unicode normalization (NFKD) converted fullwidth Chinese punctuation into standard ASCII punctuation, leaving a non-empty string even after all meaningful characters were lost. Because the code checked for an empty string to flag fully lost payee names, transactions with names like '北京,上海' slipped through as a false 'partial loss' warning instead of a 'complete loss' alert. This meant QBO files were imported with correct dates and amounts but blank payee fields, with no accurate warning shown to users. The fix replaced the empty-string check with a regex test for at least one alphanumeric character, correctly identifying strings containing only leftover punctuation as fully emptied. The case illustrates a broader pitfall: wherever Unicode normalization precedes an emptiness check, the real question should be whether any meaningful content survives, not merely whether any characters remain.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Claude Code Action Can Automatically Review GitHub Pull Requests

The anthropics/claude-code-action GitHub Action lets Claude review pull requests and answer @claude mentions in issues without a dedicated server, running on the user's own GitHub runner. It selects its behavior from the workflow context, handling code questions, PR reviews, and simple fixes, and sends API calls to a chosen provider such as Anthropic, Amazon Bedrock, Google Vertex AI, or Microsoft Foundry. Setup is done by running claude /install-github-app from a repo terminal, which requires repository admin rights and is documented for direct Anthropic API users, with other providers covered in the repo's cloud-providers guide. The project's Solutions Guide lists review patterns such as automatic PR review, path-specific checks, and OWASP-aligned security reviews, and the article recommends starting small and limiting tool access. The author stresses that the AI review should be treated as a first pass rather than a replacement for human approval, and notes they did not run the action on a live repository.

0
ProgrammingDEV Community ·

A 65B Local Model Outscored a 501B Model on Identical Eval Questions

A developer reported that a 65-billion-parameter model run locally scored 80.52 on their evaluation suite, higher than a 501-billion-parameter frontier model on the same test. Both models answered the same 200 multiple-choice questions at temperature 0 with no system-prompt changes, using the same scoring script. The local model was quantized to Q4_K_M and ran on a single consumer GPU, while the larger model was accessed through its hosted API with default settings. The smaller model reportedly won 14 of 20 categories. The author noted that verifying the larger model's result costs about twelve dollars per full run, compared with roughly forty minutes of GPU time and electricity for the local run, and shared a short video walkthrough of the setup.

0
ProgrammingDEV Community ·

Developers Explore Running Local LLMs in Browsers Using WebGPU and Transformers.js

A technical article from DEV Community describes how WebGPU gives browsers direct access to local GPU compute, enabling quantized open-weight models to run client-side. It names Qwen 2.5, Phi-3.5, and Llama 3.2 as examples of models that can execute within the browser sandbox. The author highlights a code example using the Transformers.js pipeline API and notes challenges such as shader compilation latency during initial load. Caching weights through the browser's Cache Storage API is suggested so models are downloaded only once. The piece argues that keeping prompts on the device avoids server round trips and improves data privacy for sensitive information.

0
ProgrammingDEV Community ·

Deduplizieren einer Kundenliste in Python, wenn keine Zeilen exakt identisch sind

Doppelte Einträge in Kundenlisten sind selten identische Zeilen, weil sich etwa das Bestelldatum, die Stadt oder die Schreibweise der E-Mail unterscheiden. Welche Zeilen als Duplikate gelten, hängt daher vom gewählten Vergleichsschlüssel ab, der bestimmt, wie viele Zeilen entfallen und welche erhalten bleiben. Der Artikel zeigt dies anhand einer Beispielliste mit 412 Zeilen in reinem Python und vergleicht die Ergebnisse mit dem Befehl Remove Duplicates im Spreadsheet Editor von osFoundry. Ein Schlüssel über die ganze Zeile entfernt keine Einträge, die E-Mail allein entfernt alle 38 Wiederholungen, und E-Mail plus Stadt entfernt 25. Zudem behält die übliche Konvention die erste Zeile, was bei den meisten Dubletten nicht die neueste Bestellung ist, weshalb die Sortierung vor dem Deduplizieren wichtig sein kann.