GPU capacity is the only part of modern infrastructure where the shopping experience still feels like 2008. You compare a table of accelerator names and hourly rates, pick the cheapest row that looks big enough, sign up, and discover that the instance type you chose has no availability in any region you want, that your model does not actually fit, or that the thing you rented gets reclaimed twelve minutes into a training run.
The reason is that the visible number, the hourly rate, is the least predictive part of the decision. What decides whether a GPU cloud works for you is whether the accelerator has enough memory for your weights plus activations plus KV cache, whether the provider can actually give you that accelerator when you ask rather than in three weeks, and whether you are buying raw compute or a managed inference path with all the autoscaling and cold start behaviour that implies.
Those three questions produce completely different shortlists. A team fine-tuning a large model needs multi-GPU nodes with fast interconnect and will tolerate a queue. A team serving an open-weights model behind an API needs scale-to-zero, predictable cold starts and no queue at all. A team calling a hosted model needs neither, and should not be renting GPUs in the first place.
This is how I would evaluate the category: what determines fit, what determines availability, what determines your tail latency, and where the line sits between a GPU rental and an inference product.
Key takeaways
- Accelerator memory, not raw throughput, is what decides whether a model fits. Weights are only the first term; activations, batch size and KV cache are the rest.
- Advertised capacity is marketing. Real availability varies by region, accelerator class, node size and contract term, and it is the first thing to verify in a trial.
- Renting a GPU and renting an inference endpoint are different products with different failure modes. One charges you for idle time, the other charges you for cold starts.
- Spot and preemptible capacity is genuinely cheap and genuinely interruptible. It works for training with checkpointing and is a poor fit for latency-sensitive serving.
What actually determines whether your model fits
The specification sheet gives you a name, a memory figure and a throughput figure. Only one of those is a hard constraint.
Memory capacity is the wall. A model does not fit in the memory taken by its weights. It needs the weights, the activations for the batch you are running, optimiser state if you are training, and for autoregressive generation a key-value cache that grows with sequence length multiplied by batch size. The KV cache is the term teams forget, and it is the one that turns a model that loaded fine in testing into an out-of-memory error under production concurrency. Quantisation reduces the weight term and does nothing about the cache term.
Memory bandwidth decides token throughput. For inference on a single accelerator, generation is usually memory-bound rather than compute-bound: every generated token requires reading the model weights, so tokens per second tracks bandwidth far more closely than it tracks peak arithmetic throughput. Two accelerators with the same memory capacity and very different bandwidth will give you very different serving performance on the same model.
Interconnect decides whether multi-GPU works. If a model does not fit on one accelerator you split it across several, and now the speed at which those accelerators talk to each other matters more than any of their individual specs. A node with high-speed direct GPU-to-GPU links behaves very differently from a node where GPUs communicate over the host bus. Across nodes, the fabric matters the same way. This is the single biggest hidden difference between providers that otherwise list identical hardware: two vendors can both offer eight accelerators in a box and only one of them wired them together properly.
Precision support decides which optimisations you can use. Newer accelerator generations support lower-precision formats that roughly halve memory per parameter and increase throughput. An older generation with the same memory figure cannot run the same quantised kernels, so “same VRAM” does not mean “same workload”.
Driver and framework versions decide whether any of it runs. The gap between a provider’s image and the version your framework expects is a real source of lost days. Providers that ship maintained images for common frameworks save you that time; providers that hand you a bare instance make it your problem.
Needs first-hand data: Load your actual model at your actual target concurrency and measure peak accelerator memory, not model size on disk. Record the peak at your p99 sequence length with the batch size you intend to serve. That number, not the parameter count, is what you shop for.
Advertised capacity is not available capacity
Every GPU cloud publishes a list of accelerator types. That list describes what the provider sells, not what you can get today, in your region, at the node size you need.
Availability is shaped by several things at once. The newest accelerator generation is almost always constrained, and the constraint is usually allocated to committed-contract customers before it reaches on-demand. Large multi-GPU nodes are scarcer than single GPUs because they require a whole physical machine with the right interconnect. Specific regions run out independently of each other, so “available” on a status page can mean available in one data centre on another continent. And spot or preemptible pools have their own availability, which is correlated with on-demand scarcity in exactly the wrong direction: when demand spikes, spot capacity disappears first because it is reclaimed to serve on-demand.
The practical consequences are worth internalising before you sign anything.
If you need guaranteed capacity, you are buying a reservation or a committed contract, and the negotiation is about term length and minimum commitment rather than hourly rate. If you can tolerate a queue, providers with a queueing model will serve you eventually and the queue length is the real metric. If you need burst capacity on demand with no notice, you need a provider whose business model is serving exactly that, and you will pay for the privilege in the unit price.
Multi-provider is the honest answer for anything that must not stop. Not because any provider is unreliable, but because accelerator scarcity is a market condition rather than a vendor failure, and a workload pinned to one vendor’s capacity in one region is exposed to a shortage nobody controls. This is one of the few categories where the abstraction layer earns its keep: keeping your training and serving code portable across providers costs a little discipline and buys you an actual option.
Needs first-hand data: During a trial, request your target configuration at your target size in each region you care about, at three different times of day across a week, and record whether it was fulfilled, queued, or refused. That log is worth more than any capacity claim on a pricing page.
Renting a GPU is not the same as renting an inference endpoint
This is the distinction that sorts the whole category, and the pricing pages deliberately blur it.
A GPU rental gives you an accelerator attached to a machine for a period of time. You install what you want, you run what you want, and the meter runs whether or not anything is happening. Utilisation is entirely your problem. This is the right shape for training, for fine-tuning, for batch jobs, and for serving at a volume high enough that your GPUs are busy most of the time.
An inference platform gives you a model or a container behind an HTTP endpoint and meters something closer to work done: requests, tokens, or compute-seconds while your code is actually executing. The platform handles autoscaling, queueing and scale-to-zero. This is the right shape for spiky traffic, for prototypes, and for anything where your average utilisation would be embarrassing.
A hosted model API gives you somebody else’s deployment of somebody else’s weights, metered per token. No infrastructure at all. This is the right shape for most applications until you have a specific reason it is not, and the specific reasons are usually a custom fine-tune, a data residency requirement, or unit economics at genuinely high volume.
The economics of this choice are not subtle. A dedicated GPU running at low utilisation is the most expensive way to serve inference that exists, and it is what teams accidentally build when they lift a training setup into production. A serverless inference platform running a high-volume steady workload is also expensive, because you are paying a platform margin on compute you could have rented flat. The crossover point is a function of your traffic shape, and you can calculate it from your own request pattern rather than guessing.
Traffic shape matters more than traffic volume here. Steady load favours dedicated capacity. Bursty load with long idle periods favours scale-to-zero. Diurnal load with a predictable peak favours dedicated capacity sized for the trough plus a serverless overflow path, which is more engineering but is genuinely the cheapest answer for a lot of products.
Cold start is a model loading problem
For serverless GPU inference, cold start is the number that decides whether the product is usable, and it is dominated by a step that has nothing to do with the platform’s scheduler.
The sequence is: acquire an accelerator, pull and start the container image, load the model weights into host memory, transfer them to accelerator memory, compile or warm any kernels, then serve. Weight loading and transfer is usually the largest term, and it scales with model size. A multi-gigabyte weight file has to travel from storage to accelerator memory, and how fast that happens depends on where the weights are stored relative to the machine, whether the platform caches them on local disk, and whether the platform can snapshot a warmed process rather than redoing the work.
This is why platforms differentiate on things that look like plumbing. Weight caching near the compute, container image streaming that starts execution before the whole image has landed, memory snapshotting that restores a ready process rather than reloading, and keeping a configurable number of instances warm. Those four mechanisms are the entire cold start story, and a platform that has none of them cannot be fast no matter what its scheduler does.
Two things you control matter as much as the platform. Bake weights into the image or into a cached volume rather than downloading them from a third party at startup, because a cold start that includes an internet download of several gigabytes is not going to be fast. And keep the runtime import path small, because framework imports and kernel compilation run on every cold start.
Needs first-hand data: Measure cold start as the time from an HTTP request arriving at a scaled-to-zero deployment to the first token of the response, with your own weights and your own container. Record it at p50 and p99 across at least a hundred cold invocations, separately for each candidate platform. Median cold starts hide the tail that your users experience.
Spot capacity and what interruption actually does to you
Spot, preemptible and interruptible instances are surplus capacity sold at a discount with the condition that it can be reclaimed. The discount is real and large. The question is what happens to your workload when the reclaim notice arrives.
For training, interruption is a solved problem if you solved it: checkpoint frequently to durable storage, make the job resumable from a checkpoint, and accept that wall-clock time becomes unpredictable. The cost of checkpointing is storage plus the time spent writing, and the sizing question is how much progress you are willing to lose. Teams that do this well run long training jobs almost entirely on interruptible capacity.
For serving, interruption is much worse than it looks. Losing an instance mid-request drops the request. Losing several at once removes capacity during whatever demand event caused the reclaim. The mitigation is over-provisioning across zones and providers, and at that point the discount has been partly spent on redundancy. Interruptible capacity for latency-sensitive serving is a choice you make deliberately with a fallback path, not a default.
The notice period is the detail that decides your options. A workload that gets a meaningful warning can drain connections and checkpoint; a workload that gets almost none can only be designed for abrupt loss. Notice periods differ between providers and are worth confirming rather than assuming.
CoreWeave

CoreWeave is a cloud built specifically around accelerated compute rather than a general cloud with GPUs added. That focus shows up in the parts that matter for large training jobs: high-speed interconnect between nodes, Kubernetes as the primary interface, and capacity planning oriented around multi-node clusters rather than single instances. It sits at the infrastructure end of the market, which means it is a strong fit for teams with a platform function and a poor fit for a solo developer who wants a notebook.
Pros
- Built around multi-node accelerated compute, so cluster interconnect is a first-class design concern rather than an afterthought
- Kubernetes-native interface means your existing orchestration, scheduling and observability tooling carries over
- Storage and networking are designed for training data throughput rather than general-purpose workloads
- Capacity is contracted and planned, which makes large committed workloads predictable in a way on-demand markets are not
Cons
- Oriented toward substantial committed workloads, so it is not where you start with an experiment
- Requires real Kubernetes competence to use well, which is a staffing assumption not every team can meet
- Fewer adjacent managed services and a narrower regional footprint than a hyperscaler, so databases, identity and data residency options all come from elsewhere
Best for: Teams running sustained multi-node training or large-scale inference who have platform engineers and are ready to commit to capacity.
Pricing: Sold as accelerated infrastructure with on-demand and committed-contract options, metered per instance-hour with separate meters for storage and networking. Committed terms are where the economics change.
Lambda Labs

Lambda sits in the middle of the market: dedicated GPU instances and multi-GPU clusters with a much shorter path from signup to a working machine than a hyperscaler, and framework-ready images that save you the driver and CUDA archaeology. It is the closest thing to a workstation-shaped experience delivered as a cloud, which is exactly what a research or fine-tuning team usually wants.
Pros
- Preconfigured machine learning images mean the environment is working before you start, not after a day of driver debugging
- Simple, comprehensible instance model without the surrounding complexity of a general cloud
- Offers both on-demand instances and reserved multi-node clusters, so a team can grow without changing vendors
- Persistent filesystem options make dataset handling straightforward across instance restarts
Cons
- Availability of the newest accelerator classes fluctuates, and popular configurations can be unobtainable on demand
- Thin surrounding platform and a short region list, so networking, identity, managed data services and geographic placement are all yours to arrange
- Not a serverless product, so idle instances bill at full rate and utilisation discipline is entirely on you
Best for: Research and fine-tuning teams who want a ready-to-use GPU machine quickly without operating a cloud platform around it.
Pricing: Metered per instance-hour for on-demand GPUs, with reserved and cluster arrangements priced by term. Storage is metered separately from compute.
RunPod

RunPod is a marketplace-shaped GPU cloud with two distinct products: persistent pods that behave like rented machines, and a serverless mode that scales containers with GPUs attached. The marketplace structure, with community-supplied capacity alongside data centre capacity, is what makes it unusually cheap and also what makes reliability something you have to reason about per tier rather than assume.
Pros
- Two models in one platform, so the same account covers a long-running training pod and an autoscaled inference endpoint
- Wide range of accelerator classes including older generations that are perfectly adequate and much cheaper for many workloads
- Fast to start, with a low-friction path from container image to running GPU
- Community capacity tier makes experimentation genuinely inexpensive
Cons
- Reliability and network quality vary between the community tier and the data centre tier, and the difference is material for production
- Persistent storage semantics differ between pod types and are easy to misconfigure into data loss
- Capacity for specific accelerator types in specific regions can vanish without notice
Best for: Individual developers and small teams who want cheap experimentation and a straightforward path to autoscaled inference without enterprise procurement.
Pricing: Per-second metering on pods by accelerator class, with a separate serverless meter based on active compute time. Storage for persistent volumes is billed separately.
Modal

Modal is a serverless compute platform where GPUs are one of the resources a function can request. You write Python, decorate a function with the hardware and container image it needs, and the platform handles building, scheduling, autoscaling and scale-to-zero. Its engineering investment has gone into container and snapshot startup, which is precisely the mechanism that decides whether serverless GPU inference is viable.
Pros
- Container and memory snapshot machinery is aimed directly at the cold start problem rather than working around it
- Infrastructure defined in the same Python as the application, so there is no separate deployment language to maintain
- Scale-to-zero with per-second metering means idle costs nothing, which transforms the economics of spiky workloads
- Handles batch jobs, scheduled work and web endpoints in one model, so a pipeline and its serving path live in one place
Cons
- Python-centric and opinionated by design, so another language is an awkward fit and porting an existing containerised service is a reshaping exercise
- Long-running, permanently busy workloads pay a platform margin over renting equivalent capacity directly
- Less control over the exact machine and network topology than a raw instance, which matters for tightly tuned multi-node training
Best for: Python teams serving bursty inference or running batch AI pipelines who want autoscaling and scale-to-zero without operating a cluster.
Pricing: Metered per second of active compute by resource class, with no charge while scaled to zero. Storage and egress are metered separately.
Together AI
Together AI operates at both layers: a hosted inference API for open-weights models, and dedicated GPU clusters for training and fine-tuning. That combination makes it useful as a progression path, since you can start against a shared endpoint, fine-tune on the same platform, and move to dedicated capacity when volume justifies it without changing your integration surface.
Pros
- One vendor covers hosted inference, fine-tuning and dedicated capacity, so the migration path is a configuration change
- Serving stack is tuned for open-weights models, which is meaningful work you do not have to replicate
- Fine-tuned models can be served on the same platform, removing a deployment step most alternatives leave to you
- OpenAI-compatible API surface for inference keeps client code portable
Cons
- Hosted inference means your model selection is limited to what the platform has deployed
- Dedicated cluster availability and terms are a sales conversation rather than a self-service action
- Per-token pricing on shared endpoints can be hard to forecast until you know your real token distribution
Best for: Teams building on open-weights models who want hosted inference now and a credible path to fine-tuning and dedicated capacity later.
Pricing: Per-token metering for hosted inference by model, separate metering for fine-tuning jobs, and contracted pricing for dedicated GPU clusters.
Replicate

Replicate packages models as containers behind a uniform HTTP API, with a large public catalogue and the ability to push your own. The abstraction it sells is that running a model should be an API call with no infrastructure, and it holds up well for image, audio and video models where the operational overhead of self-hosting is disproportionate to usage.
Pros
- Very large catalogue of ready-to-run models across modalities, so evaluating an approach takes minutes
- Uniform API across wildly different models removes per-model integration work
- Custom models package as containers with a defined interface, which is a low-ceremony path to production
- Scale-to-zero by default, so an occasionally used model costs nothing between calls
Cons
- Cold starts on infrequently used models are a real part of the user-visible latency and vary with model size
- Per-second compute metering on long-running generative jobs can add up faster than a reserved instance at steady volume
- Public model quality and maintenance vary, since the catalogue is community-contributed
Best for: Product teams integrating image, audio or video models who want an API rather than an inference platform to operate.
Pricing: Metered per second of compute time by hardware class, with scale-to-zero meaning no charge between requests. Some catalogue models are priced per unit of output instead.
Fal

Fal is an inference platform focused specifically on generative media, with heavy engineering on making diffusion and related model families fast. The specialisation is the point: a platform that optimises one workload class deeply will beat a general one on that class, and generative media has enough latency sensitivity in interactive products that the difference is user-visible.
Pros
- Optimised specifically for diffusion and generative media workloads rather than being general-purpose
- Streaming and real-time oriented APIs suit interactive products where latency is part of the experience
- Custom model deployment is supported alongside the hosted catalogue
- Client libraries and queueing primitives are built around long-running generation rather than request-response
Cons
- Narrow focus means it is the wrong tool for language model serving or general training
- Smaller catalogue than the broad model marketplaces, and no answer for training or batch work if you wanted one vendor
- Optimisation work is specific to supported architectures, so an unusual custom model may not see the same benefit
Best for: Products built around image, video or audio generation where latency is part of the user experience and you do not want to tune a diffusion serving stack.
Pricing: Metered per request or per second of compute depending on model, with dedicated capacity available for sustained volume.
AWS
AWS offers accelerated instances across several generations, managed training and inference through its machine learning platform, and its own silicon for training and inference workloads. The argument for it is rarely that it is the best GPU cloud and almost always that it is where everything else already lives: your data, your identity model, your network boundary and your compliance posture.
Pros
- Co-located with your existing data, which removes the egress and latency cost of moving training sets between clouds
- Existing identity, networking and compliance controls extend to accelerated workloads with no new access model
- Multiple purchasing models including spot, savings plans and capacity reservations give real flexibility on commitment
- Custom training and inference silicon offers a different price and availability curve from the general accelerator market
Cons
- Capacity for the newest accelerator classes is constrained and frequently requires a reservation or a quota conversation
- Considerably more configuration surface than a focused GPU cloud, so time to first working job is longer
- Cost model spans several meters at once, and accidental cross-zone or egress charges are easy to incur
Best for: Organisations whose data, compliance boundary and platform already sit on AWS, where moving the workload to the data beats moving the data to a cheaper GPU.
Pricing: Per-instance-hour metering by accelerator type with spot, on-demand, reservation and savings plan options, plus separate meters for storage, data transfer and managed platform features.
Google Cloud
Google Cloud offers GPU instances alongside its own tensor accelerators, with a managed platform covering training, tuning and serving. Its distinguishing asset is the alternative accelerator line, which has a genuinely different availability and performance profile from the mainstream GPU market and is well supported by the frameworks that target it.
Pros
- An alternative accelerator family with separate availability, which is a real hedge when GPU capacity is scarce
- Managed platform covers training, tuning and endpoint serving under one identity and networking model
- Strong data and analytics adjacency, which matters when training data already lives in the platform
- Multiple commitment models including spot and reservations for capacity planning
Cons
- Getting full value from the alternative accelerators requires framework and code adaptation, which is real engineering effort
- GPU quota and regional availability require planning rather than being available on request
- Egress and cross-region transfer charges apply to moving large datasets and model artefacts
Best for: Teams whose data already sits in Google Cloud, and teams willing to adapt code to an alternative accelerator in exchange for a different availability curve.
Pricing: Per-instance-hour metering for GPUs and per-chip-hour for the alternative accelerators, with spot and committed-use discounts, plus separate meters for storage and network egress.
How to choose
Answer these in order and the shortlist collapses quickly.
Are you training or serving? Training wants dedicated multi-GPU capacity with fast interconnect and tolerates queues and interruptions if you checkpoint. Serving wants availability, predictable cold start and autoscaling. Very few providers are the best answer to both, and trying to pick one vendor for both usually means compromising the one that matters more.
Does your model fit on one accelerator? If yes, your options are wide and you are shopping on availability and price. If no, interconnect quality becomes the dominant criterion and the shortlist narrows to providers who built for multi-node work.
Where is your data? Training data movement is expensive and slow. If your dataset is measured in terabytes and sits in one cloud, the cheapest GPU somewhere else may not be cheaper once you have paid to move the data. The same logic that governs object storage and egress applies here and is usually decisive.
What is your traffic shape? Steady, high-utilisation serving favours dedicated instances. Spiky traffic with long idle windows favours serverless inference. If you cannot describe your traffic shape yet, start serverless, because the failure mode of over-provisioned dedicated GPUs is an expensive bill and the failure mode of serverless is a slow first request.
Do you need a GPU at all? If you are serving a standard open-weights model at moderate volume, a hosted API is cheaper and simpler than anything in this list. Rent GPUs when you have a specific reason: a custom fine-tune, a residency requirement, or volume high enough that the per-token margin exceeds the operational cost.
Whatever you pick, do three things in week one: measure peak accelerator memory at real concurrency rather than trusting parameter counts, log capacity request outcomes so you know your real availability, and keep your serving container portable so a capacity shortage at one vendor is a redeploy rather than a redesign.
| Provider | Shape | Strongest when | Main risk |
|---|---|---|---|
| CoreWeave | Infrastructure | Sustained multi-node training | Commitment and Kubernetes overhead |
| Lambda Labs | Instances | Fast path to a ready GPU machine | Availability on popular classes |
| RunPod | Instances and serverless | Cheap experimentation, simple autoscale | Tier-dependent reliability |
| Modal | Serverless | Bursty Python inference and batch | Platform margin at steady high volume |
| Together AI | API and dedicated | Open-weights serving plus fine-tuning | Model selection limited to the catalogue |
| Replicate | Model API | Multimodal models as an API call | Cold starts on infrequent models |
| Fal | Model API | Interactive generative media | Narrow workload focus |
| AWS | General cloud | Data and compliance already there | Capacity constraints and cost surface |
| Google Cloud | General cloud | Data already there, alternative accelerators | Code adaptation for non-GPU silicon |
Frequently asked questions
How do I know which accelerator my model needs?
Start from memory, not throughput. Add the memory taken by weights at your chosen precision, the activation memory for your batch size, and for generative models the key-value cache at your p99 sequence length and target concurrency. Measure the total under real load rather than calculating it, because attention implementations and serving frameworks differ substantially in their overhead. Only once several accelerators clear that bar does bandwidth become the thing you compare.
Is serverless GPU inference cheaper than a dedicated instance?
It depends entirely on utilisation. Serverless wins decisively when your GPUs would otherwise sit idle, which covers most products before they find real traffic. A dedicated instance wins when utilisation is consistently high, because you stop paying a platform margin on every compute-second. Calculate your crossover point from your own request logs: multiply your requests per hour by your average compute-seconds per request and compare that to the hours you would rent.
What happens when my spot instance is reclaimed?
You get a notice of some length, then the instance goes away. For training, this is survivable with frequent checkpointing to durable storage and a resumable job, and it is how most cost-conscious teams run long jobs. For serving, in-flight requests are lost and capacity disappears at the worst possible moment, so interruptible serving needs on-demand fallback capacity and health checking that reacts fast. Confirm the notice period with your provider before designing around it.
Should I run inference on my own GPUs or call a hosted API?
Call the hosted API until you have a concrete reason not to. The reasons that actually justify self-hosting are a custom fine-tune the API cannot serve, a data residency or privacy constraint, a latency requirement that a shared endpoint cannot meet, or volume where the per-token margin clearly exceeds your operational cost. Prototyping on a hosted API and moving later is cheap; starting with your own GPUs and discovering you did not need them is not. If you route across several model providers, an AI gateway keeps that decision reversible.
How many providers should I plan for?
At least two if the workload cannot stop. Accelerator scarcity is a market condition, not a vendor reliability problem, and a shortage at one provider in one region is a normal event. The cost of portability is keeping your training and serving code in containers with no vendor-specific glue, which is good practice anyway. The same argument applies to multi-region deployment generally, and it is stronger here because the supply constraint is real.
Related reading
- Best hosting platforms for developers — the broader hosting landscape and where accelerated compute fits into it.
- Best serverless function platforms — the same scale-to-zero and cold start mechanics without the accelerator.
- Best container hosting platforms — where a portable inference container runs when you are not renting GPUs.
- Best AI gateways — routing, fallback and cost control across model providers.
- Best LLM observability tools — measuring what your inference path actually does in production.
- Best S3 alternatives and object storage providers — where training data and model weights live, and what moving them costs.