SShortSingh.
Back to feed

PySBD vs yasbd-lib: How Two Python Libraries Approach Sentence Boundary Detection

0
·4 views

Two Python libraries — PySBD and yasbd-lib — tackle sentence boundary detection with fundamentally different architectures. PySBD, available since 2020 and supporting 22 languages, works by temporarily replacing punctuation with placeholder tokens before splitting text, then restoring the original content. This mutable approach means character-offset spans must be reconstructed after processing, which can become computationally expensive on long or repetitive documents. In contrast, the newer yasbd-lib supports 39 languages and treats the input string as immutable, tracking index pointers throughout detection rather than modifying the text at all. As a result, yasbd-lib produces character spans natively during parsing, avoiding the post-processing overhead that PySBD requires.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Azure Cosmos DB vNext Emulator Gains AI Agent Support via CLI and Skills

Microsoft's Azure Cosmos DB vNext emulator now supports AI coding agents such as GitHub Copilot CLI, Codex, and Claude Code through its built-in Cosmos DB Shell, an open-source CLI that runs inside the emulator container. The CLI's command-based structure makes it well-suited for agent workflows, allowing agents to discover operations, execute commands non-interactively, and act on returned results. A dedicated Agent Skill called cosmosdb-emulator-vnext packages task-specific guidance to help agents inspect resources, create databases and containers, load test data, and run queries. In a demonstration, a developer prompted GitHub Copilot CLI to set up a local emulator with synthetic multi-tenant order data, and the agent autonomously created a partitioned database with 12 realistic orders spanning multiple statuses. The integration aims to reduce manual setup work, letting developers describe desired outcomes rather than writing out individual shell commands.

0
ProgrammingDEV Community ·

Software Engineer Rebuilds Career After Burnout, Illness, and Layoff

Cesar Aguirre, a software engineer with over 10 years of experience, is publicly sharing his journey of career rebuilding after facing serious personal and professional setbacks. He experienced illness, burnout, and a layoff, all of which significantly disrupted his career path. Aguirre has turned to writing as a form of therapy and is also working on a book trilogy as part of his recovery and reinvention process. He opened an Ask Me Anything (AMA) session on DEV Community on July 20, inviting others to engage with his story. The post sparked discussion around mental health, resilience, and career challenges in the tech industry.

0
ProgrammingDEV Community ·

Claude for Chrome and Tap MCP Servers Offer Complementary Browser Automation Roles

Claude for Chrome is a browser assistant that uses live AI reasoning to handle one-off tasks, exploration, and judgment-dependent workflows inside a user's logged-in Chrome session. However, because it re-reasons through every task on each run, repeated workflows consume model tokens and require the machine to remain active. Tap, a separate MCP server running alongside Claude for Chrome in the same Claude Code session, addresses this by compiling a completed browser flow into a deterministic .tap.json plan. Once compiled, the plan replays the workflow at zero LLM token cost on a local schedule, without needing further AI inference. The two tools are designed to complement each other: Claude for Chrome handles novel or adaptive tasks, while Tap handles the repetitive execution of already-solved workflows.

0
ProgrammingDEV Community ·

How systemd-run Makes Linux cgroup Resource Limits Simple and Instant

Linux developers dealing with runaway processes can use systemd-run to apply memory and CPU limits without editing service files or rebooting. The command wraps the cgroups interface managed by systemd, letting users set constraints like a 256MB memory cap directly at runtime. For persistent limits, service unit files support directives such as MemoryMax and CPUQuota to enforce hard resource ceilings on long-running services. The MemoryHigh directive offers a softer threshold that triggers kernel memory reclamation before a hard limit is reached. Developers can inspect live cgroup data via the /sys/fs/cgroup filesystem, making it easier to monitor and debug resource usage without modifying system configuration.