RunAnywhere open-sources TrnRT, an LLM inference engine for AWS Trainium
Startup RunAnywhere, a Y Combinator W26 company, has released TrnRT (Trainium RunTime) as open source, an inference engine built for AWS Trainium chips. The engine currently runs the GLM-5.3 flash model in FP8 precision on a single trn2.48xlarge instance, serving two model instances with a cache-aware router. The team says it reached 3.57 million tokens per minute on SemiAnalysis's AgentX benchmarks and used custom NKI kernels to speed up decoding. The company began the project because memory on Trainium is cheaper than on comparable Nvidia H200 setups, and it found that AWS's own inference frameworks lacked support for the model. The project is open to contributions, and the team plans to add more models.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in