What Does Your Reverse Proxy Cost?

This is the second post in the Campfire series. The first one fixed posting. This one is about the read pages, where the largest change came from outside the C# code.

The numbers here were measured during development, one change at a time, on development machines. They aren’t the final benchmark.

The setup

Basecamp’s Campfire image runs Thruster, a small reverse proxy written in Go, in front of the Rails app. Thruster handles TLS and Let’s Encrypt certificates, HTTP/2, X-Sendfile, serving files from public/, caching responses marked public, and the X-Forwarded-* headers. The C# port is meant to be a drop-in for that image, so it shipped the same way: Thruster in front, Kestrel behind it, both in one container.

The Rust, Go and C ports don’t. They serve HTTP from their own process.

The symptom

After the posting fix, C# read pages at two and a half to almost four times Rails. That’s respectable, but it’s low for compiled C# against Ruby.

The container had four CPUs, and two processes were sharing them. Every request was parsed by Thruster, sent over a loopback connection to Kestrel, parsed again, answered, and the response copied back through the proxy. All of that work came out of the same four CPUs the app needed.

Measuring the hop

The test was to run the same image twice, once as shipped and once with Kestrel answering directly, on the same machine and harness on the same day. Development numbers from the i9-11900K, four CPUs for the app, requests per second:

Route Through Thruster Kestrel directly  
Room page 6,481 19,996 3.1×
Messages page 9,176 27,895 3.0×
Sidebar 9,872 30,515 3.1×
Search 9,364 30,164 3.2×
Post a message 1,449 1,392 about the same

The proxy capped every read route at about a third of what the app could serve. Posting didn’t change, because posting is limited by the single SQLite writer, not by the CPUs.

Jose Luis Chavez del Cid (@jlchavez) measured the same thing on his own machine (Docker Desktop, four CPUs per app) and put a number on the hop itself: Thruster spent about 0.5 to 0.8 ms of CPU on each proxied request. The C# app spent about 448 microseconds on a whole page, so the proxy cost more than the app did. His health-check route went from 4,352 to 20,094 requests a second without it.

Rails pays the same toll, but it hardly shows. A Rails request costs about 2 ms of CPU, so another half a millisecond is a small share. The faster the app behind the proxy, the larger the proxy’s share of each request.

The lesson

A reverse proxy costs a fixed amount of CPU per request, whatever app is behind it. When the proxy runs on the same CPUs as the app, that cost comes straight out of the app’s throughput. As your app gets faster, the proxy becomes a bigger fraction of the total, until it’s the main cost.

Most of the time this is a good trade. A proxy in front of a slow app costs almost nothing in relative terms, and it does real work. But it’s worth knowing the number, because it decides how much faster your app can get before the proxy is what you’re measuring.

What the proxy was doing for you

Removing the proxy means something else has to do its jobs. For Campfire that list was:

  • TLS, and getting and renewing certificates.
  • HTTP/2.
  • X-Sendfile, so large files aren’t streamed through the app.
  • Serving public/ files and the error page shown when the app is down.
  • Setting X-Forwarded-*.
  • Caching responses that say they can be cached publicly.

The last one caught us out. Campfire’s avatars controller answers with expires_in 30.minutes, public: true. Thruster kept those responses, so a repeat request for an avatar never reached the app. With Thruster gone, every avatar request did a query and a file read, and avatars became the one route where C# was slower than Rails. Jose fixed it in pull request #256 with an in-memory AvatarCache that keeps each avatar for those 30 minutes, with Thruster’s own limits of 64 MB in total and 1 MB per item. Authentication, the ETag check and every header are unchanged.

So the lesson has a second half: before you remove a proxy, list what it does, including the things you never configured.

What we shipped

The image still runs Thruster by default, because a self-hosted install needs TLS and certificates. Jose’s pull request #253 added a CAMPFIRE_THRUSTER=0 switch that runs the app alone, for people who already have nginx, Caddy or a load balancer in front, and for measuring the app without the hop. That’s how C# runs in the scoreboard, and the debrief’s fine print says so: Rails was measured behind Thruster, as it ships, and C# without it, as the Rust, Go and C ports run.

The Docker post shows both ways of running it.

Check your own app

  • Measure your app with and without the proxy, on the same machine, with the same number of CPUs.
  • Work out the proxy’s CPU per request and compare it with your app’s. If they’re close, the proxy is a large part of your bill.
  • If the proxy and the app share a container or a small VM, they share CPUs. A proxy on its own machine, or a managed load balancer, doesn’t take from the app.
  • Before you remove a proxy, write down everything it does, including caching, compression, header rewriting and static files.

Try it in LINQPad

This script starts a tiny ASP.NET Core app and a second server that forwards every request to it, then measures both. Enable “Reference ASP.NET Core” in the query properties (F4). The load generator runs in the same process, on the same CPUs, which is the situation the container was in.

// The app: one cheap endpoint, like a health check.
var appBuilder = WebApplication.CreateBuilder();
appBuilder.Logging.ClearProviders();
var app = appBuilder.Build();
app.Urls.Add("http://127.0.0.1:5101");
app.MapGet("/up", () => "OK");
await app.StartAsync();

// The proxy: a second server that forwards every request to the app.
var upstream = new HttpClient { BaseAddress = new Uri("http://127.0.0.1:5101") };
var proxyBuilder = WebApplication.CreateBuilder();
proxyBuilder.Logging.ClearProviders();
var proxy = proxyBuilder.Build();
proxy.Urls.Add("http://127.0.0.1:5102");
proxy.Run(async context =>
{
    using var response = await upstream.GetAsync(context.Request.Path);
    context.Response.StatusCode = (int)response.StatusCode;
    await response.Content.CopyToAsync(context.Response.Body);
});
await proxy.StartAsync();

await RequestsPerSecond("http://127.0.0.1:5101/up", seconds: 2); // warm up both
await RequestsPerSecond("http://127.0.0.1:5102/up", seconds: 2);

new
{
    Direct = await RequestsPerSecond("http://127.0.0.1:5101/up"),
    ThroughProxy = await RequestsPerSecond("http://127.0.0.1:5102/up"),
}.Dump("Requests per second, 16 clients");

await proxy.StopAsync();
await app.StopAsync();

async Task<double> RequestsPerSecond(string url, int clients = 16, int seconds = 5)
{
    using var client = new HttpClient();
    var stopAt = DateTime.UtcNow.AddSeconds(seconds);
    long count = 0;
    await Task.WhenAll(Enumerable.Range(0, clients).Select(async _ =>
    {
        while (DateTime.UtcNow < stopAt)
        {
            using var response = await client.GetAsync(url);
            response.EnsureSuccessStatusCode();
            Interlocked.Increment(ref count);
        }
    }));
    return Math.Round(count / (double)seconds);
}

On my machine the proxied endpoint served about half as many requests as the direct one. This toy proxy is simpler than a real one, and the endpoint does almost nothing, so it’s the extreme case: the cheaper your endpoint, the more the hop matters.

The next post goes back inside the app, to a query that ran several times on every request.

csharp dotnet performance kestrel reverse proxy thruster docker linqpad