Big Allocations, Small Gains

This is the sixth post in the Campfire series. The last one found a gain three times larger than predicted. This one is the opposite: two fixes that looked enormous in the profile and turned out small.

The numbers were measured during development, one fix at a time, on an Intel i9-11900K running Linux with the app on four CPUs. They aren’t the final benchmark.

Two allocations that dominated the profile

An 80 KB buffer for an empty body

The first allocation profile of the room page had one site far ahead of the rest. RequestBody.ReadAllAsync, which reads a request body into memory, created an 80 KB chunk and a MemoryStream before its first read. A GET has no body, so the read returned nothing straight away, but the buffers had already been allocated. That was 83% of all bytes allocated under load, about 1 GB a second at 7,400 requests a second.

The fix was small: rent the chunk from ArrayPool<byte>.Shared, and create the MemoryStream only when the first byte arrives. An empty body now allocates nothing.

This change landed together with the first response cache fix, and the two together added 7 to 12% on the read pages. We didn’t measure the buffer fix on its own, so I can’t tell you its share. The cache fix removed a lock that was keeping half the CPUs idle, which makes me think most of that gain was the lock.

A 415 KB page copied into a growing buffer

Later, with several other fixes in, the allocation profile had a new leader. The room page is 415 KB of HTML. The app buffers each response so that compression and a couple of header-adding steps see the whole body, and it did that by copying the finished page into a fresh MemoryStream. A MemoryStream grows by doubling, so every request reallocated its buffer step by step up to 512 KB, and the larger steps landed on the large object heap.

MemoryStream.set_Capacity was 5.85 GB of the 21.3 GB sampled in 8 seconds, 27% of all sampled bytes and the largest single site. The process was allocating about 2.5 GB a second at 99% CPU.

The fix hands the controller’s finished body to the pipeline as it is, with no copy. Anything still written to the buffer, such as an error page, follows it, so the bytes on the wire don’t change. A test checks one rendered body sent plain, gzipped and as a HEAD request.

After the fix, set_Capacity was gone from the profile, and sampled byte[] allocation fell from 7.66 GB to 1.48 GB per 8 seconds.

The result

Route Quick A/B Benchmark harness
Room page +3.1% −0.5%
Messages page +1.3% +2.9%
Sidebar +2.1% +0.8%
Search +1.6% +2.9%

That’s one to three percent for removing 80% of the sampled byte[] bytes, and the harness numbers are inside its run-to-run noise.

Even that number needed care. The .NET SDK image moved from 10.0.12 to 10.0.13 between the previous build and this one, so a plain comparison would have mixed a runtime change into the result. We rebuilt the previous commit on the new runtime and used that as the control. Whenever you A/B two container images, check that the base image didn’t move underneath you.

Why bytes weren’t the cost

Allocating a large buffer in .NET is cheap per byte. The runtime hands out memory and zeroes it, which runs at close to memory bandwidth. What you pay for later is collection, and that was small here: with DATAS off, Jose Luis Chavez del Cid (@jlchavez) measured the process paused in the GC for 2.2% of wall time at benchmark load. If GC is a few percent of your time, removing a large share of your allocations can only win back part of those few percent. The copy was real work, 415 KB of memory traffic per request, but copying memory is one of the fastest things a CPU does.

Compare that with post 7 in this series. Jose cut the allocation per request from 51 KB to 29 KB, a much smaller number of bytes than either fix here, and gained about 10%. His allocations were small strings, each one built by searching an environment block, running a regex, escaping, parsing or serializing JSON. The bytes were a side effect of CPU work, and removing the allocation removed the work.

The two results don’t contradict each other. Bytes allocated is a good signal for finding code that does work on every request, and Jose’s method of measuring bytes per request is the steadiest yardstick we found. It’s a poor way to rank how much each site costs. An allocation profile sorts by bytes, and a big, cheap allocation will always top that list.

The lesson

Before you spend a day removing an allocation, ask what produced the bytes. A large buffer that’s allocated and filled once is probably cheap. Many small objects, each the output of parsing, formatting or encoding, are probably expensive, and the CPU profile will show the work even when the allocation profile ranks them low.

Then look at your GC time. If the process spends 2% of its time in the GC, no allocation fix can give you more than about 2% from GC savings alone. The rest has to come from the work you remove along with the allocation.

Check your own app

  • Read your GC pause percentage, from dotnet-counters or your APM, before ranking allocation fixes.
  • For each top allocation site, ask what work produces the bytes. A buffer copy and a string built by a parser cost very different amounts per byte.
  • Look for buffers created before you know you need them, such as a read buffer for a body that may be empty.
  • Don’t copy a finished response into a MemoryStream to buffer it. Hand over the bytes you already have, or pre-size the stream.
  • When you A/B container images, pin the base image, or rebuild the control on the same one.

Try it in LINQPad

This script measures bytes allocated and time taken for a few things a request might do, and divides one by the other. Add System.Buffers and System.Text.Json to the query’s namespaces.

var session = new { Id = 42, Token = "f3a9c1e07b", ExpiresAt = DateTimeOffset.UtcNow };
var cookieHeader = "theme=dark; session_token=f3a9c1e07b--4d2a; _campfire_session=abc123%3D%3D--9f8e";

new[]
{
    Measure("A new 80 KB buffer", () => _ = new byte[81920]),
    Measure("An 80 KB buffer from the pool", () => ArrayPool<byte>.Shared.Return(ArrayPool<byte>.Shared.Rent(81920))),
    Measure("Split and unescape a cookie header", () =>
    {
        foreach (var pair in cookieHeader.Split("; ")) _ = Uri.UnescapeDataString(pair.Split('=', 2)[1]);
    }),
    Measure("Serialize a small JSON document", () => _ = JsonSerializer.Serialize(session)),
}.Dump("Per call");

object Measure(string name, Action action, int iterations = 100_000)
{
    for (var i = 0; i < 1_000; i++) action(); // warm up
    var bytes = GC.GetAllocatedBytesForCurrentThread();
    var clock = Stopwatch.StartNew();
    for (var i = 0; i < iterations; i++) action();
    var allocated = (GC.GetAllocatedBytesForCurrentThread() - bytes) / iterations;
    var nanoseconds = clock.Elapsed.TotalNanoseconds / iterations;
    return new
    {
        Name = name,
        BytesAllocated = allocated,
        Nanoseconds = Math.Round(nanoseconds),
        NanosecondsPerByte = allocated == 0 ? (double?)null : Math.Round(nanoseconds / allocated, 3),
    };
}

On my machine the 80 KB buffer allocated 110 times the bytes of the cookie parse and took about 10 times as long. Per byte, the cookie parse cost 10 times as much as the buffer, and the JSON serialization about 100 times as much. Sort these four by bytes and then by nanoseconds per byte, and you get the two orders this post is about.

The next post is Jose’s, and it’s where the small allocations turn out to matter.

csharp dotnet performance allocations gc arraypool memorystream linqpad