Direct Browser Uploads of Large Training Files Hinge on Multipart Part Size

The article argues that the size of artifacts a platform accepts, rather than the upload protocol itself, should drive the design of large-file ingestion. It recommends presigned multipart uploads sent directly from the browser to object storage, with a signing service that only issues signatures, and choosing part size from the largest expected object. The author notes that object stores cap multipart uploads at 10,000 parts with a 5 MiB minimum part size and a 5 TiB object limit, so a poorly chosen part size can cause failures late in an upload. Part size also trades request costs against retry costs, with a 16 to 64 MiB range common for artifact workloads. The piece compares three paths, proxying through the API, presigned multipart, and self-hosted resumable uploads, and concludes that presigned multipart is clearly preferable for artifacts above a few hundred megabytes.