Python's float output inflates LLM token costs, and short timestamps and rounding can cut them sharply
A developer found that a day of sensor data exported with Python's csv module used about 49,700 tokens, far more than the information requires. Long floating-point values like 19.944444444444446 are split into many tokens by the tokenizer, but the timestamp column was the bigger cost, making up 58% of the file. Shortening timestamps to HH:MM and rounding values to two decimals reduced the file to about 17,300 tokens, a 65% saving. Rounding to one decimal lost real measurements, so the author recommends rounding only to the precision the source actually has. The article includes a script that compacts CSV files before they are sent to a model.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in