Student Builds Tiny Japanese GPT in PyTorch and Tests a Toy Self-Improvement Loop
A second-year ZEN University student built a small character-level Japanese language model from scratch in PyTorch, using only basic tensor operations and no pretrained weights. The decoder-only Transformer has roughly 0.9 million parameters and was trained in three stages: text continuation on short excerpts of modern Japanese literature, fine-tuning on dialogue data to act as a chatbot, and a best-of-N self-training loop. In that loop, the model generates several candidate answers per prompt, scores them with a rule-based function covering completion, formatting, keyword relevance and its own confidence, and retrains on the highest-scoring pairs over three generations. The author stresses that the model is an educational toy trained on a few kilobytes of text, not a useful system, and that the loop's score gains should not be read as genuine intelligence improvement. The project also includes a browser-based interface that displays next-character probabilities and allows retraining.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in