GPU cloud vendors — Runpod, Modal, and the rest of the serverless-and-dedicated-GPU cohort — don't bill per token at all, and the mechanics of per-second or per-GPU-hour pricing have a few non-obvious gotchas that a straight per-token comparison misses entirely. Here's what's actually driving the number on the invoice.
Per-second billing rounds up, and idle time still counts
Runpod's own pricing documentation is explicit that serverless billing is per-second, rounded up to the next full second, and that idle time counts too — a worker sitting warm during its idle-timeout window (5 seconds by default before scaling to zero) is billed at the standard compute rate for that entire window, whether or not it's actively processing a request. A workload with frequent, small requests spaced just under that idle window can end up paying for far more idle seconds than active ones.
Cold starts and model loading are billable time, not free setup
Spinning up a worker and loading a model onto it takes real time, and that time is billed the same as inference time — it isn't a free setup phase. Runpod offers two specific mitigations: FlashBoot, which speeds up cold starts for supported configurations, and model caching, which avoids re-downloading the same model weights on every cold start. Neither eliminates the cost of a cold start entirely, but both shrink the billable window it occupies.
Some vendors bill the higher of what you requested or what you used
Modal's documentation describes a different mechanic: billing is per-second, but based on the maximum of the resources you requested and what you actually consumed — so under-provisioning doesn't save money if a burst pushes actual usage above the request, and over-provisioning does cost you even if actual usage stays low. Modal also allows a workload to burst above its requested CPU or memory using spare host capacity, billing only for what was actually used during the burst rather than a pre-reserved ceiling. Exact idle-timeout and rounding specifics vary by vendor and workload type and are worth checking directly against current docs before assuming Runpod's numbers apply elsewhere.
Always-on capacity trades a higher floor for eliminating cold starts
Alongside scale-to-zero serverless pricing, Runpod offers an always-on Active tier — dedicated capacity running 24/7 with negotiated discounts available through sales — for workloads where a cold start on every request isn't acceptable. That's the same trade-off that shows up across the GPU-cloud category generally: pay for guaranteed, always-warm capacity, or pay per-second for capacity that scales to zero but reintroduces a cold start each time it spins back up.
Normalize to your actual workload before comparing to a per-token API
A raw per-GPU-hour or per-second rate tells you nothing on its own — the number that actually matters is cost per completed request, which depends on how many requests a given GPU configuration can serve per hour at your typical prompt and output length. That figure has to be worked out per model and per workload; there's no shortcut that converts a GPU-hour rate directly into a per-million-tokens number without knowing your own throughput. Check the calculator for whichever GPU-cloud vendor you're evaluating for its current verified rate, and add a free watch so a rate or billing-model change reaches you before it shows up as a surprise.