Claude Builds a Small Language Model From an Empty Folder, With Mixed Results

An experiment gave Claude an empty folder and no budget, tasking it with building a language model that could beat GPT-2 small using only free Kaggle hardware. Claude designed its own tokenizer, data pipeline and training recipe, and recorded predictions and pass marks before each run. Its first level was generating children's stories, where it added a custom 'narrative clock' idea that improved endings by about 7%. The first full run produced only four of ten required passing stories, falling short of its predicted seven. A retry with a larger model also missed its target, and the author ultimately chose to move on from that level.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.


Discussion (0)
Log in to join the discussion and vote.
Log in