Bot-Generated Articles Skew Wikipedia Word Frequencies in Low-Resource Languages
An analysis of word frequency lists from several Wikipedia editions suggests that some minority-language versions are unreliable as linguistic datasets. In balanced human-written text, function words like prepositions and conjunctions dominate, but Tatar and Bashkir Wikipedias are topped by content words such as "river" and "water" because many articles were auto-generated from templates using state water registry data. Erzya and Moksha Wikipedias show a similar distortion, with frequent terms tied to bot-created orchid stubs, a pattern the author calls the Orchid Phenomenon. Average article lengths in these editions are also much shorter than in Russian Wikipedia, raising the risk of repetitive template-driven content. The author advises developers to profile corpora with simple frequency counts before training language models on Wikipedia dumps.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in