Design Sets the Ceiling

This post pulls together the Campfire series. Posts 1 to 7 each took one fix from the C# port and turned it into a lesson. Looking at them side by side, the biggest ones have something in common: they weren’t about C#.

The per-fix numbers here were measured during development, one fix at a time, mostly on an Intel i9-11900K running Linux with the app on four CPUs. They aren’t the final benchmark.

Where the big numbers came from

Sort the fixes by size and the top of the list is design:

The checkpoint trigger (post 1). Posting went from 50 to 1,449 messages a second, from a fifth of Rails to almost six times Rails, in the same language with the same code around it. The change was one pragma: count WAL frames instead of watching a file that never shrinks.
The proxy hop (post 2). Taking the proxy out of the path tripled every read page. Jose Luis Chavez del Cid (@jlchavez) measured the proxy at 0.5 to 0.8 ms of CPU per request, more than the C# app spent on a page. That cost doesn’t depend on the language behind the proxy. Rails pays it too, and at about 2 ms per request it hardly notices.
The cache check (post 3). Asking SQLite “has anything changed?” with a query, several times per request, inside one lock. It took three steps to fix. The last two alone added about 6 to 10% and then another 14 to 24% to the read pages.

The next tier is about how the code used the platform: logging that every request paid for (post 4), crypto contexts rebuilt on every call (post 5), and the small strings Jose found (post 7). These are worth 10 to 20% each. The large allocations in post 6 were worth 1 to 3%.

So the order was: how many processes and hops a request goes through, then how many queries and locks it takes, then what each line of code costs.

What the fastest port does differently

The C port of Campfire is fastest on the room page, and it holds up best when writes keep emptying the caches. Reading its source, the reasons are mostly design, and most of them are decisions about the same things this series fixed.

One campfire executable replaces Ruby, Puma, Redis, Resque and Thruster. Its own front end speaks TLS with ALPN and HTTP/2, so a request is parsed once, in the process that answers it.
The HTTP server and the WebSocket server each run their own epoll loop.
The complete-response cache has one mutex, and the version it checks under that mutex is a number in memory. Observing the database happens outside the lock.

Several of the C# fixes copied these ideas. Observing data_version on pooled connections outside the mutex came straight from the C port. The WAL-index header read went one step further for a reason that’s also about design: the C port’s in-process counter doesn’t see commits made by other processes, and the C# port has to, because Rails does. The C port also doesn’t support existing installs or schema upgrades. The C# port is a drop-in for the Rails app, so it keeps Rails’ schema, storage layout and cookies. Some of the work that costs C# time is work the C port chose not to do.

Where the language comes in

Once the design problems were gone, Jose profiled the room page with Linux perf, so native code showed up alongside managed code. At about 24,000 requests a second with the app’s four CPUs at 95 to 99%:

Share of CPU samples	Where
24%	The .NET base class libraries: strings, spans, dates, JSON, collections
19%	The app’s own code
19%	The kernel (sockets, `epoll`) and addresses perf couldn’t name
12%	The .NET runtime: allocation, GC, type checks, stubs
9%	OpenSSL: the cookie HMAC and AES-GCM, SHA-256
8%	ASP.NET Core and Kestrel
6%	libc and the loader
4%	JIT stubs

The app’s own code is a fifth of the CPU, and no function in it is above about 2%. The profile is flat. What’s left is the cost of doing the work in .NET: the runtime, the libraries, the web server and the crypto library underneath. That share belongs to the language and its runtime, and it decides how close you get to the ceiling the design sets.

There’s still room. Jose’s notes list what the profile suggests next, in order of size and risk: a prefilter for routing, which runs a regular expression for each candidate route; a file lookup before routing that costs 2.4% on requests that are never files; the cookie’s Base64 and date parsing; and parsing the trusted proxy list once instead of on every request. Each is worth a few percent, and none of them is a design change.

The scoreboard

The final run put every port on one Google Cloud VM with four cores each, on the shared harness, twice from scratch. All 196 million measured responses were checked against the route contracts first, and none were invalid. Requests per second, each route on its own:

Workload	Rails	C#	Rust	Go	Express	Laravel	C
Room page	2,388	37,115	42,167	28,555	20,880	1,999	58,248
Messages page	2,183	71,154	40,027	29,344	40,521	2,062	60,615
Sidebar	2,407	78,359	48,775	32,711	49,147	2,283	64,849
Search	2,491	76,385	48,822	33,112	46,937	2,171	66,091
Post a message	265	2,598	3,198	3,264	1,228	463	1,729

C# is fastest on the messages, sidebar and search pages, 16 to 21% ahead of C and 56 to 78% ahead of Rust. It’s third on the room page and posts at about ten times Rails, a little behind Rust and Go.

The weak spot shows up with a writer posting in the background. C#’s room page drops to about 13,000 requests a second, fifth of the seven, while C keeps 52,000. The other three pages stay at or near the top. My guess is that every write clears the response cache, and the room page is the most expensive page to rebuild, so it pays the most. I haven’t measured that yet, and if it’s right, it’s another design problem rather than a language one.

The full tables, both sessions, p99 latencies and every raw sample are in bench/PORTS.md. Rails runs behind its Thruster proxy as it ships; the other ports, C# included, serve HTTP from their own process.

What I’d tell myself at the start

Count the hops before you profile. How many processes does a request pass through? How many times is it parsed? Does anything else share the CPUs? On the read pages the proxy cost us more than any single code fix, and you can see it on an architecture diagram.

Count the queries and locks per request, including the fast ones. The cache check was a fast query that ran five times per request inside a lock every request needed.

Check that background work stays in the background. A maintenance job whose trigger never resets ends up in the hot path as soon as the load is steady.

Then profile, and fix one thing at a time. Each of the code fixes was found by a profiler, proved with an A/B, and checked against the Rails app’s responses before it counted. Several of my guesses about what was slow were wrong, in both directions.

Expect the language to matter last. Moving from Ruby to C# didn’t make posting fast. Posting was slower than Rails until the checkpoint was fixed. Once the design was right, the language and runtime decided what was left.

The last two posts in the series are about running the C# port yourself: in Docker, and inside a LINQPad script.