Tiny 11.9M-Parameter Transformer Learns to Generate Web Apps After Simple Data Fixes

A researcher built a small 11.9-million-parameter transformer, trained on 4,176 generated programs, to convert plain-text feature specs into working Python web applications. Despite correct architecture and sufficient training, the model scored zero compilable outputs across five consecutive experiments, which the researcher initially misattributed to insufficient model capacity. The actual cause turned out to be two simple data pipeline flaws: docstrings were tokenised as single atomic tokens bloating the vocabulary to nearly 6,000 entries, and training samples lacked end tokens so the model never learned to stop generating. Fixing both issues — splitting docstrings into words and adding end tokens — reduced vocabulary size from 5,953 to 1,817 entries and immediately lifted compile success from 0 out of 6 to 6 out of 6. The researcher notes that the production tool still uses rule-based compositional synthesis rather than the model, and this post serves as a public retraction of an earlier claim that turned out to be premature.