Death by a Thousand Strings: Small Per-Request Costs in .NET
Everything in this post was found and fixed by Jose Luis Chavez del Cid (@jlchavez). He took the profiling loop from the Campfire port, pointed it at the code after our big fixes were done, and found what we had walked past: a dozen small costs that every request paid. His four pull requests cut the memory allocated per request from 51 KB to 29 KB and added roughly 10% throughput on every page. He also wrote down exactly how he measured it, in bench/PROFILING.md, which is the best guide to profiling this app that exists.
When there’s no hotspot left
By the time Jose started, the obvious problems were gone. The checkpoint, the proxy, the lock, the logging and the crypto contexts were fixed. The room page ran at about 25,000 requests a second with the app’s four cores at 99%, and the CPU went to the code every signed-in request shares, with no single function standing out.
That’s where most tuning stops, because there’s nothing obvious left to point at. Jose switched yardsticks instead.
Measure bytes per request, not requests per second
Throughput is noisy. Two runs of the same image differed by 3 to 8% on his machine, so a 3% improvement disappears in the noise. Allocation is steadier:
bytes per request = allocation rate ÷ requests per second
Both numbers come from the .NET runtime counters (dotnet-counters, or the dotnet-diagnostics MCP server Jose used). Read over 15 seconds of steady load, after a warm-up, it repeats to within about 3%. So a change shows up in bytes per request long before a throughput run can prove it.
Two habits from his notes are worth stealing:
- Warm up first. For its first 10 to 25 seconds under load, the app served about half its steady rate while the JIT recompiled the hot code. Anything measured then is measuring the JIT.
- Alternate the order. Run A, B, B, A rather than A, A, B, B, so that drift on the machine (heat, background work) lands on both sides equally.
Then he asked the allocation profiler who was allocating, and fixed them one at a time.
The thousand strings
None of these would make a list of “performance mistakes” on its own. Each one is ordinary, readable C#. The problem is that each ran on every request, often several times.
An environment variable read on every call
The response cache read its size limit from CAMPFIRE_RESPONSE_CACHE_MB every time it needed it, about five times per cached request. Environment.GetEnvironmentVariable isn’t free: it searches the environment and allocates a new string each time. It was 4.5% of the CPU samples taken while requests were being handled.
A running process’s environment doesn’t change, so read it once:
// Before: every call searches the environment and allocates a string
static long Budget => ParseMb(Environment.GetEnvironmentVariable("CACHE_MB"));
// After: read once, when the type is first used
static readonly long budget = ParseMb(Environment.GetEnvironmentVariable("CACHE_MB"));
A regular expression to find a port
To work out the request’s base URL, the app found the port at the end of the Host header with /:(\d+)$/, up to three times per call, and several parts of the request asked for the base URL. The regex engine alone was 1.5 to 1.9% of samples.
The fix is to compute host, port and base URL once per request and read the port from the end of the string by hand: find the last colon and check that only digits follow. Jose kept the exact behavior of the regex, including what \d and $ accept, so the output couldn’t change.
JsonNode for a three-field document
Every response writes two cookies, one encrypted and one signed, each wrapped in the same small JSON envelope: a message, an expiry and a purpose. Building that with JsonNode means a dictionary per object, a node per value and a serializer pass, for a document whose shape never changes. Reading it back did the same in reverse.
Jose wrote the common case straight into the output bytes, encoding and Base64 in place, and read it back as offsets into the original bytes. Anything unusual (an escape, a different key order, a non-ASCII byte) still goes to the old parser, so the result is identical for every input. On its own this cut 6 KB from every request.
Strings built from strings
The cookie code escaped, parsed and formatted text through StringBuilder, string.Split and intermediate strings. The response cache key ran each piece through a JSON encoder and sorted the cookies with LINQ’s OrderBy. Each step made a string to hand to the next.
The fixes follow one pattern: return the input untouched when nothing needs changing (most cookie values need no escaping), build the result in a stack or pooled buffer when something does, and sort a handful of items with Array.Sort and an ordinal comparer instead of a LINQ pipeline.
What it added up to
From Jose’s A/B runs, each measured against the same baseline image on the same machine:
| Change | Allocated per request | Room page | Messages | Sidebar | Search |
|---|---|---|---|---|---|
| Before | 51.2 KB | ||||
| Cookie text without intermediate strings | 40.7 KB | +4.8% | +2.2% | +2.0% | −0.7% |
| Cache budget read once, cache key in one buffer | 47.1 KB | +11.4% | +13.4% | +9.6% | +11.2% |
Cookie envelope without JsonNode |
44.9 KB | ||||
| All three together | 30.4 KB | +12.6% | +9.2% | +10.7% | +11.3% |
| Plus host, port and base URL computed once, measured against the three together | 29.1 KB | +9.6% | +11.1% | +5.0% | +3.7% |
The single-change throughput figures are noisy, so the “all three together” row is the one to trust. The cookie envelope’s throughput wasn’t measured on its own, only its allocation. None of these changed a byte of any response. Every A/B validated each response against the Rails app first.
Why this matters more than it looks
Earlier in this series, we removed two huge allocations, an 80 KB buffer and a 415 KB page copy, and gained 1 to 3%. Jose removed much smaller ones and gained about 10%. That looks like a contradiction until you count the work instead of the bytes. A large buffer rented once and copied once is cheap. A dozen small strings, each built by searching, parsing, encoding or sorting, are CPU work first and garbage second. Bytes allocated per request is still the right thing to measure, because it’s steady and it points at the code doing the work.
Try it in LINQPad
You don’t need a load test to see this. GC.GetAllocatedBytesForCurrentThread() counts what the current thread has allocated, so you can measure any snippet directly:
long Measure(Action action, int iterations = 100_000)
{
action(); // warm up
var before = GC.GetAllocatedBytesForCurrentThread();
for (var i = 0; i < iterations; i++) action();
return (GC.GetAllocatedBytesForCurrentThread() - before) / iterations;
}
var host = "chat.example.com:8443";
var portRegex = new Regex(@":(\d+)$", RegexOptions.Compiled);
new
{
EnvironmentRead = Measure(() => Environment.GetEnvironmentVariable("PATH")),
RegexPort = Measure(() => portRegex.Match(host).Groups[1].Value),
SpanPort = Measure(() => int.Parse(host.AsSpan(host.LastIndexOf(':') + 1))),
}.Dump("Bytes allocated per call");
Run it and compare the three lines. Then point Measure at the code your own requests run every time.
Where it stopped
After the first three changes, Jose profiled again with Linux perf, so native code showed up too. This time the profile really was flat: the app’s own code was about a fifth of the CPU, and no function in it was above about 2%. The rest was the .NET libraries, the runtime, ASP.NET Core, OpenSSL and the kernel. That’s a good signal that the app’s own per-request waste is mostly gone, and it’s the subject of the next post in this series.
The lesson
When a profile goes flat, the remaining cost is spread across code that looks fine. Change the yardstick to bytes per request, which is steady enough to show small wins, and look for work that runs on every request but only needs to run once: reading configuration, parsing the same header again, or building a general-purpose object for data whose shape never changes.
Thank you, Jose, for finding all of it, and for writing down how.