Developer built a padding-free deep learning framework in 2018 that foreshadowed modern LLM serving

In 2018, a master's student built InsNet, a C++14 deep learning library designed to eliminate batch padding by storing all values in a flat buffer with per-sequence shape metadata. The library was motivated by NLP tasks like tree-LSTMs and hierarchical encoders, where each input had a unique computation graph shape, making traditional padding wasteful and complex. Around 2021, the developer publicly claimed padding-free dynamic batching was a critical missing feature in existing deep learning libraries, but the idea gained little traction at the time. By 2023, vLLM popularized 'continuous batching' for LLM inference, using a flat, padding-free token stream with per-sequence offsets — structurally identical to InsNet's core design. The author argues the 2018 approach was not wrong but simply early, and targeted the wrong layer of the stack compared to where the industry eventually adopted the concept.