NLP course post explains tokens, word types, and why 'word' is hard to define

·3 views

An introductory natural language processing post opens its series by asking what counts as a word. It uses a single example sentence to separate word instances from distinct word types and the vocabulary. The author notes that spoken filler, punctuation, capitalization, and morphology all complicate the count. Formal definitions and Heaps' law describe how vocabulary grows with corpus size, which motivates splitting text into subword units.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Passkeys Aim to Replace Passwords With Device-Based Login

Long-standing advice to use stronger passwords has failed to address problems like reuse and easy guessing. Passkeys remove the password altogether by pairing a private key stored on the user's device with a public key held by the website. Logins are confirmed through a fingerprint, face scan, or screen lock, so no shared secret crosses the internet to be intercepted. Major platforms including Apple, Google, and Microsoft now support them, though syncing, setup, and recovery can still be inconvenient. For now, many sites keep passwords as a fallback during a gradual transition.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Large Language Models Like ChatGPT Work: A Plain-English Explainer

Chatbots such as ChatGPT generate human-like replies by repeatedly predicting the most likely next token, which are small chunks of text. The model learns these patterns through pre-training on vast amounts of text, storing what it learns as numerical parameters, and is then fine-tuned on helpful conversations to act more like an assistant. Most modern models use the transformer design, whose attention mechanism weighs how earlier tokens relate to the next one. Because outputs are built from plausible patterns rather than verified facts, these systems can state false information confidently, a problem known as hallucination.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Public Wi-Fi Risks Today: Fake Hotspots and Login Traps Matter More Than Eavesdropping

Long-standing advice to avoid public Wi-Fi entirely is outdated, since most sites now use encryption that protects the content of what users send and view. The main remaining dangers include fake hotspots set up to mimic legitimate venue networks, phony sign-in pages that harvest passwords, and captive portals requesting excessive personal information. Exposure can also come from devices with file sharing enabled or outdated software that is easier to attack. Experts recommend verifying network names with staff, using apps instead of browser logins for sensitive services, marking networks as public, keeping devices updated, and considering a VPN or personal phone hotspot.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Tools Are Making Restaurant Menus Easier to Navigate for Diners

Reading a restaurant menu can be difficult for people managing dietary needs, allergies, or health goals, since traditional menus often lack detailed nutritional information. A new field called AI menu analysis uses natural language processing, computer vision, and machine learning to extract and enrich menu data. These tools can estimate calories, macronutrients, micronutrients, and allergen content, sometimes even from dish images. Combined with user profiles, the technology can highlight suitable dishes, such as vegan options or safe choices for nut allergies, and flag less suitable ones. The article, published on DEV Community, presents this as a way to reduce the cognitive load of comparing dishes on large menus.

0 comments Read more at DEV Community