RTX 3090 vs L40S: Relative Speed Depends on Batch Size, Study Finds

A developer measured an RTX 3090 against an L40S on rented machines using the same script, testing 7B fp16 models. The 3090 matched the L40S at batch size 1 (1.09x) but reached only 0.46x its speed at batch size 128. The author concluded there is no single speed figure, since relative performance depends on how many requests run concurrently. Cost was also factored in, with the 3090 rented at $0.414/h and the L40S at $0.826/h. The author noted that a different 3090 unit tested earlier was slower across all settings, so hardware should be measured per unit. All 192 runs and the script are published on GitHub.