Developer Releases 150 GB Open-Source Dataset for Central Asian Languages and Code

·16 views

A developer has published a nearly 150 GB multilingual dataset on Hugging Face to address the severe shortage of open-source training data for Central Asian languages, including Kyrgyz, Kazakh, Uzbek, and Tajik. The dataset also includes 20 GB of source code in Python, C++, Rust, and Go, making it one of the few resources combining low-resource languages with technical corpora. Building the dataset took weeks of collecting, parsing, and cleaning data, while compressing the archive to around 27.7 GB caused repeated out-of-memory crashes and took roughly 10 hours to complete. The project is released under the CC BY 4.0 license, allowing free use for research, benchmarking, and commercial purposes. It is intended to support tasks such as code LLM fine-tuning, technical translation, and continual pre-training of lightweight language models tailored to Central Asian contexts.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Android 'no internet' fault traced to rogue IPv6 router advertisements from a 5G router

A user found that their Android phone showed Wi-Fi as connected but without internet, while other devices worked normally. After isolating the problem, they traced it to a Huawei 5G router (CPE) that was broadcasting IPv6 router advertisements despite being set to LAN-only mode. Disconnecting the device made the phone's issue disappear, and reconnecting brought it back. The author explains that Android's connectivity check depends on reaching its servers, and that stray IPv6 gateway announcements can redirect the phone's traffic into a dead end. The article describes how to detect these advertisements using rdisc6 or tcpdump on the same network segment.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Debugging interviews reward investigation process over a lucky first patch

Live debugging rounds on broken UIs or unfamiliar repos are evaluated on how the candidate investigates, not on how quickly they reach a working fix. Recommended practice is to restate expected versus actual behavior, identify the layer where the fault likely sits, form one testable hypothesis, and run the cheapest probe before editing code. Guides from several interview-prep sources emphasize this sequence over rewriting components or making silent, trial-and-error changes. Interviewers tend to watch how candidates narrow the search space, and a patch that hides the symptom without finding the root cause can score poorly. Some employers restrict AI or external tools during these rounds, so candidates are advised to confirm the rules with the recruiter beforehand.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why the key skill in AI-driven engineering is knowing when to reject AI proposals

AI tools can now propose architectures, split work into tasks, write code, and prepare deployments, prompting new methodologies such as AWS's AI-Driven Development Lifecycle. The author argues that the decisive question is who actually makes the engineering judgments, not how fast AI can generate output. In this view, the most valuable human contribution is the authority to reject proposals, set constraints, request alternatives, and halt execution. Without that authority, human oversight risks becoming a ceremonial approval step. The piece illustrates the danger with an example of an agent proposing an overly complex design, such as adding Kafka, Redis, and a new microservice for a simple reservation feature.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Home intrusion detector's syn_scan rule could never fire, author reports

A developer building CNDS, a home-network intrusion detector that uses Scapy and five detection engines, ran it on a Raspberry Pi 5 and saw alert volumes of around 45 per hour on day one, with roughly 178 and 336 false positives on two October days. Causes included untrained demo models saturating their scores, critical payload regex rules firing on encrypted traffic, and a monitoring probe generating most remaining alerts. After fixing these, daily alerts dropped to 37 to 63 with none rated high or critical. A red-team test then revealed that the syn_scan rule had a contradictory condition that made it impossible to trigger, since every SYN packet also counts as a forward packet. The author fixed the rule and made it critical, though a SYN flood test still produced no alert, prompting further investigation.

0 comments Read more at DEV Community