Buyer’s Guide

Serverless Cold Starts Compared

Written by Govind Kumar Lohar. Reviewed for technical accuracy by Deepak Gupta and Bhaskar Suthar on · Review panel

  • serverless
  • benchmark
  • lambda
  • hosting

Independent buyer’s guide. No vendor paid to be included, ranked or described a particular way. Written for engineers, architects and the people who sign off on their tooling budget. Editorial policy.

This article does not contain cold start numbers. That is deliberate, and it is worth being direct about why.

Every cold start comparison you can find publishes a table of milliseconds per platform. Almost none of them publish the harness, the deployment configuration, the package size, the region pairs, the sampling method, or how they forced a cold start in the first place. Without those, a number is not a measurement. It is an anecdote with a decimal point, and it cannot be reproduced, defended or applied to your workload. Several of those tables also compare platforms that meter different clocks, which makes the comparison meaningless before the first invocation.

I have not run this benchmark. Rather than publish figures I cannot stand behind, what follows is the methodology: precisely what to measure, what must be held constant, how to force cold starts reliably, what the platform-reported numbers do and do not include, and the specific mistakes that void a result. The results section is a specification of what to fill in, not a table of invented values.

This is more useful than a number, for a reason worth stating. Your cold start is dominated by your own deployment package and your own initialisation code, not by the platform. A published figure for a hello-world function tells you almost nothing about the function you are actually going to deploy. A methodology tells you how to get the number that applies to you, which is the only one that matters.

Key takeaways

  • A cold start is not one thing. On container-based platforms it is sandbox creation plus runtime boot plus your initialisation. On isolate platforms there is no sandbox to create and the cost is script evaluation. Comparing them as one number compares different phenomena.
  • Platform-reported init duration excludes things your users experience, including scheduling delay and, in a virtual private network configuration, network attachment. Client-observed latency includes things that are not cold start. You need both.
  • Cold start latency is bimodal by definition, so means are misleading. Report the cold and warm distributions separately, each at the median and the tail.
  • The variables that must be held constant are region, memory setting, runtime version, deployment package size, provisioned or pre-warmed capacity, and time since last invocation. Varying any one of them silently invalidates the comparison.

What a cold start actually is, per platform

The term covers at least three different mechanisms, and treating them as one number is the first error.

Container-based function platforms

The classic case. When no warm execution environment is available, the platform must create one. That involves, roughly in order: a scheduling decision about where to place it, creation of the isolation boundary, fetching or mounting the deployment package, starting the language runtime, running any code outside your handler, and then invoking the handler.

Two of those phases are worth separating carefully. Runtime boot is the language starting: a virtual machine warming up and loading classes, an interpreter starting and importing modules, or a static binary being executed, and the spread between those is enormous. Initialisation code is everything at module scope in your own package: constructing clients, opening pools, reading configuration, building caches. This is the phase you control, and in most real functions it is the largest one.

There is an additional phase that platform-reported metrics may or may not include: network attachment. A function placed inside a private network needs a network interface, and the creation of that interface has historically been a significant and separately-billed-in-time part of the cold path. Whether your measurement captures it depends on where you measure.

Isolate platforms

There is no sandbox to create in the conventional sense. The runtime is already running and your script is loaded into a fresh isolate, so the cost is dominated by parsing and evaluating your module graph. That scales with bundle size and with how much work happens at the top level of your modules.

The consequence for benchmarking is that the phrase “cold start” means something structurally different here, and a comparison table that puts an isolate platform’s script evaluation time in the same column as a container platform’s full environment creation is comparing a thing to a different thing. Report them in separate columns and say why.

MicroVM platforms

Some platforms boot a lightweight virtual machine per instance. That is a kernel boot, which is structurally more work than creating a container namespace and structurally less than booting a general-purpose virtual machine, and modern implementations use snapshotting to skip most of it. Where a platform resumes from a snapshot rather than booting, what you are measuring is snapshot restore plus whatever your application does on resume, which is again a different mechanism wearing the same name.

Scale-to-zero container platforms

Here the cold path includes the image pull if the image is not cached on the target machine, which is typically the largest single term and scales with image size. Because caching is opportunistic and invisible to you, this phase is bimodal in its own right: sometimes the image is there and sometimes it is not. A benchmark that does not report the cache-hit and cache-miss cases separately is averaging two different things. The mechanism is covered further in container hosting platforms.

Needs first-hand data: For each platform under test, record the phase breakdown separately rather than only the total: scheduling delay, sandbox or image acquisition, runtime boot, module evaluation, and first handler execution. Where the platform does not expose a phase, say so explicitly in the results rather than folding it into an adjacent one.

The harness

The harness must be boring, reproducible and hostile to its own results. Here is the design.

Deployment is code, not clicks. Every function under test is deployed from an infrastructure definition checked into the same repository as the benchmark, with the region, memory setting, runtime version, timeout and concurrency configuration all explicit. A configuration set through a console is a configuration you cannot reproduce or publish.

One image or package, built once per workload. The same source, the same lockfile, the same build command. Where platforms require different packaging formats, the differences must be documented as a limitation, because the package is a variable and you are now unable to hold it perfectly constant.

The client runs outside the provider under test. Measuring a function from inside the same cloud measures an unrepresentative network path. Run the client from a neutral location, and run it from several, because the network path between the client and the platform’s entry point is part of what you are measuring and it is not the same for every platform.

Every invocation carries a request identifier, and the function echoes back what the platform tells it. The function returns its own view of the world: whether this execution environment is new, the platform’s reported initialisation duration if one is available, the instance or isolate identifier, the region it believes it is in, and a monotonic timestamp at handler entry. That echo is what lets you reconcile client-observed latency against platform-reported duration.

Both clocks are recorded, always. Client-observed wall-clock latency from request start to last byte, and the platform’s own reported duration and init duration. The gap between them is the part of the cold path the platform does not attribute to your function, and it is frequently the most interesting number in the whole exercise.

Raw observations are stored, not summaries. Every invocation is a row. Aggregation happens at analysis time so it can be redone differently, which it will need to be.

The harness is published with the results. If it is not published, the result is not a measurement.

Needs first-hand data: Publish the harness repository alongside the results, including the infrastructure definitions, the build commands, the client runner and the analysis scripts. State the exact runtime versions and the date range over which data was collected, because platforms change and a result without a date is a result without a meaning.

The workloads

One workload is not enough, because different workloads exercise different phases of the cold path. Four are the minimum.

The floor. A handler that returns a constant string, with no dependencies at all. This measures the platform’s own overhead and nothing else, and it is the only workload where cross-platform comparison is close to fair. It is also the workload that has no bearing on any real application, which is why it must never be the only one.

A realistic dependency tree. An HTTP handler with a router, a validation library, a logging library and a cloud provider SDK. This is where deployment package size starts to matter and where the difference between a tree-shaken bundle and a directory of dependencies becomes visible.

A database-connected handler. Opens a connection or acquires one from a pool during initialisation, runs one trivial query, returns. This measures the phase that dominates real functions and almost never appears in published benchmarks. It also exposes the interaction between the concurrency model and connection establishment, which is a real operational concern rather than a curiosity.

A slow-initialising runtime. A runtime with a genuinely expensive startup, deployed both with and without whatever snapshot or pre-warming mechanism the platform offers. The delta between those two is the single most actionable number in this whole benchmark for teams on such a runtime.

Each workload should be deployed at more than one memory setting where the platform has one, because memory allocates CPU proportionally on most container platforms and initialisation is CPU-bound. A benchmark at one memory setting is a benchmark of one point on a curve, and the curve is the useful artifact. That mechanism is explained in serverless function platforms.

Needs first-hand data: Run each of the four workloads at a minimum of three memory settings per platform that exposes one, and report cold start duration as a curve against memory rather than as a single figure. Note the setting at which the curve flattens, which is where additional memory stops buying initialisation speed.

The variables that must be controlled

Each of these, left uncontrolled, produces a number that looks fine and means nothing.

Region. Both the function’s region and the client’s location. Cold start behaviour differs by region because capacity and hardware generations differ, and client location changes the network component of observed latency. Fix both, state both, and test more than one pair.

Memory setting. On most container platforms this is the CPU dial. Comparing a function at one memory setting on one platform against a different setting on another is not a platform comparison.

Runtime and version. Not just the language but the specific version, and whether it is the platform’s managed runtime or a custom image. These are different code paths with different startup costs.

Deployment package size. State it in bytes for every function under test. Package size affects the acquisition phase directly and it is the variable most often left unreported. A bundled, tree-shaken artifact and an unbundled dependency directory containing the same logic are different experiments.

Provisioned or pre-warmed capacity. Whether any is configured must be stated, because with it configured you are measuring a warm path and calling it a cold one. Test both configurations deliberately rather than leaving it ambiguous.

Time since last invocation. This is the variable that matters most and is controlled least. Platforms keep environments alive for an unpublished and variable period. A benchmark that invokes every few minutes is measuring warm starts with occasional cold ones; a benchmark that waits long enough between invocations is measuring genuine cold starts. Choose an idle interval, justify it, hold it constant, and verify from the function’s own echo that the environment really was new.

Deploy recency. The first invocations after a deployment often behave differently from invocations of the same code hours later, because caches upstream are cold too. Either wait a fixed period after deploying before collecting, or treat deploy recency as a reported dimension.

Time of day and day of week. Shared infrastructure has load patterns. Collect across at least a full week and report the spread, because a benchmark run once on a Sunday morning is a sample of one condition.

Concurrency and burst shape. A single request arriving at an idle platform and a burst of many simultaneous requests arriving at an idle platform are different experiments. The second measures scaling behaviour as well as cold start, and it is closer to what production does. Test both, separately, and never mix them in one dataset.

Network path and TLS. Whether connections are reused between measurements changes observed latency substantially. Decide whether you are measuring with a fresh connection each time or a reused one, state it, and be aware that fresh connections include a handshake that has nothing to do with cold start.

Needs first-hand data: For the time-since-last-invocation variable specifically, run a sweep: invoke at a range of idle intervals and plot the probability that the environment is new against the interval. That curve is platform-specific, is not documented by any vendor, and is more operationally useful than the cold start duration itself, because it tells you how often your users will actually experience one.

Mistakes that make a cold start benchmark meaningless

Every one of these appears in published comparisons.

Measuring means across a bimodal distribution. Cold and warm invocations are two distinct populations. Their mean lands in a region where no invocation actually occurs. Separate the populations using the function’s own report of whether the environment was new, then report each at the median and the tail.

Not verifying that a cold start happened. Waiting a while and assuming the environment is gone is not verification. The function must report whether it is a new environment, and invocations that turn out to be warm must be excluded from the cold dataset rather than quietly averaged in.

Comparing platform-reported duration across platforms. Each vendor defines and instruments its own duration differently, and some exclude phases others include. Platform-reported figures are useful for comparing a function against itself across configurations. Cross-platform comparison needs client-observed latency, with the network component understood.

Measuring from inside the same provider. It flatters that provider’s network path and produces a comparison of network topology dressed up as a comparison of runtimes.

Ignoring the gateway. An HTTP-triggered function behind an API gateway includes the gateway’s own latency, and the gateway may itself have a cold path. If the benchmark is about the function runtime, invoke the function directly. If it is about the user experience, include the gateway and say so, because these are different questions with different answers.

Using a hello-world function as the headline number. It measures the platform floor and is the least transferable result in the set. It belongs in the data as a baseline and never as the summary.

Running on one day. Shared infrastructure varies by load, by hardware generation and by ongoing platform changes. A single run is a snapshot of conditions you cannot describe.

Failing to state the date. These platforms change continuously. A cold start figure without a collection date is unusable within months, and most published tables carry figures whose provenance is now unknown.

Conflating cold start with the connection establishment that follows it. A function that opens a database connection during initialisation has a cold start that includes a network round trip to the database, which is a property of your architecture rather than of the platform. Both numbers are worth having; presenting the combined figure as a platform result is not.

Results

No results are published yet, because this benchmark has not been run. What follows specifies exactly what will go here, so that the gap is a defined gap rather than an omission.

Needs first-hand data: For each platform and each of the four workloads, publish cold and warm invocation latency separately, each at the median and at the tail, from at least two client locations, alongside the platform-reported initialisation duration where one exists, the deployment package size in bytes, the memory setting, the runtime version, the region, the idle interval used to force the cold start, and the collection date range. Report the number of observations per cell and the proportion of attempted cold invocations that turned out to be warm and were excluded.

Needs first-hand data: For the slow-initialising runtime workload, publish the same figures with and without each platform’s snapshot or pre-warming mechanism enabled, and report the delta as the primary result. This is the number that changes an architecture decision, and it is not available anywhere reliable.

Needs first-hand data: For the burst experiment, publish the distribution of latency across a simultaneous burst of invocations arriving at an idle platform, reported as a distribution rather than as a single figure, with the proportion of the burst served by new environments stated. This measures scaling behaviour rather than single-invocation cold start and is closer to what a traffic spike actually does.

Needs first-hand data: Publish the idle-interval sweep described above as a probability curve per platform, showing the likelihood that a given interval results in a new execution environment.

Until those exist, treat any published cold start table, including any that might appear here later, with the same question: can I reproduce this, and does the configuration resemble mine? If the answer to either is no, the number is not usable.

How to choose

You do not actually need this benchmark to make most decisions. Work through these instead.

Find out whether cold starts affect your users at all. Look at your production latency distribution and identify what proportion of requests are served by new execution environments. Many teams worrying about cold starts have a cold start rate low enough that it does not touch their tail latency, and many teams not worrying about it have a high rate on a low-traffic but user-facing endpoint. Measure before optimising.

Fix your own initialisation first. Deployment package size, lazy client construction, not opening connections at module scope, not fetching remote configuration before serving, not running migration checks at boot. These are almost always larger than the platform difference and they are entirely within your control.

Then use the platform’s own lever. More memory where memory buys CPU, a snapshot mechanism where one exists, pre-warmed or provisioned capacity where the latency requirement justifies always-on cost.

Only then consider changing platform. And if you do, note that an isolate platform removes this problem class structurally rather than reducing it, at the price of the runtime constraints described in edge compute platforms.

Platform classWhat “cold start” meansLargest phasePlatform leverYour lever
Container function platformEnvironment creation plus runtime boot plus initRuntime boot or your init codeMemory setting, snapshot, provisioned capacityPackage size, lazy init
Function inside a private networkThe above plus network attachmentNetwork attachment historicallyPlatform networking modeAvoid it where possible
MicroVM platformKernel boot or snapshot restoreSnapshot restore plus app resumeSnapshot supportApplication resume work
Scale-to-zero container platformImage pull plus container start plus app bootImage pull on a cache missMinimum instance countImage size, layer ordering
Isolate platformScript parse and module evaluationYour module graphNone neededBundle size, top-level work

The honest summary is that cold start is a smaller factor in platform choice than the execution model, the concurrency behaviour and the billing meter, all of which are covered in serverless function platforms and in the wider hosting platforms overview. Cold start is the thing that gets benchmarked because it is easy to benchmark, not because it is the thing that decides most architectures.

Frequently asked questions

Why are there no numbers in this article?

Because I have not run the benchmark, and publishing figures I have not measured would make everything else here worth less. The category is full of tables whose provenance is unknown, whose configuration is unstated and which cannot be reproduced. A methodology you can run against your own workload is more useful than a number from someone else’s hello-world function, because your cold start is dominated by your deployment package and your initialisation code rather than by the platform.

How do I reliably force a cold start?

Have the function report whether its execution environment is new, using whatever signal the runtime exposes, and only count invocations where it says yes. Deploying a new version guarantees new environments but conflates deploy recency with cold start. Waiting is the usual approach, and the correct idle interval is not published by any vendor and varies, which is why the idle-interval sweep is part of the methodology rather than an assumption baked into it.

Should I use provisioned or pre-warmed concurrency?

It works, and it is capacity you pay for whether or not it is used, on top of per-invocation charges. It is the right answer when you have a real latency requirement on a spiky workload and have already exhausted the cheaper levers: smaller package, lazier initialisation, more memory where memory buys CPU, and a snapshot mechanism if the runtime supports one. If it ends up covering most of your steady traffic, you have crossed the point where an always-on instance is cheaper and simpler.

Do edge and isolate platforms have cold starts?

Not in the same sense. There is no sandbox to create, so what remains is parsing and evaluating your script, which is fast but not free and scales with bundle size and top-level work. An import that builds a large lookup table at module scope is a cold start cost on an isolate platform exactly as it is on a container platform. The difference is that the floor is much lower and there is no memory dial to tune, so your bundle is the whole lever.