Under Pressure

Writing up a server that stores and serves JSON objects isn't hard. If you want it to be secure, performant, and resilient under load, things get more interesting.

Every server has its capacity limits, and to make a reliable server, you need to know what the limits are, what the bottleneck is, and how the system behaves when it's saturated. You'll also want to make it so that its behaviour degrades gracefully as the pressure mounts.

In particular, there's a scenario that's so common that it has a name: the thundering herd. This is when all of your clients will try to reach your server at once. It typically occurs after an outage: they've recognised that the server is unreachable and are all trying to ping it to reconnect, and when it comes back online, there's a massive spike in traffic – maybe ten times your usual peak load.

If the system hasn't been designed and tested for this, a very likely outcome is that the pressure spike will just crash the server. If it's in a monitored environment, the process monitor will then automatically restart it, only for it to keep crashing every time the herd thunders in again.

To avoid this scenario, the system needs to be built with back pressure. This means that it needs to recognise when it's nearing its limits, and tell its clients to back off to even the peak. TCP/IP, the networking protocol the Internet is built on, already has this at a low level – a server won't serve packets faster than a client can consume them, and vice versa – but usually this isn't sufficient to get the job done, as it only knows about network packet production and consumption.

TCP/IP back pressure would work great if requests got slower in relation to server load: more load, slower processing, slower responses, natural rate limit. In real life, this is usually not the case: once the server starts nearing its saturation point, some requests will just start to time out altogether while others go through, and when it passes the saturation point, it might just crash altogether.

That brings us to the other important question: what is the scarce resource, exactly? Network connections? CPU? Memory? Disk I/O? Database connections? Something else? While it's possible to make an educated guess (hint: it's usually database connections), there's no way to know without actually testing it. This means running a stress test on the server and monitoring its behaviour as you crank up the pressure.

With Tarinoi, we're addressing these questions using a stack of open-source components from Grafana Labs. Specifically, we're using Prometheus to scrape various diagnostics from the system, Grafana to present them in a dashboard, and k6 to run a variety of stress-test scenarios on it. I made a k6 script that simulates {secret target number} of concurrent users and measures I/O, response times, failure counts, and a bunch of other things.

A couple of runs determined that the bottleneck is, indeed, database connections (and flushed out a bug in an index!). Then I can run the test and look at the database connection graph. Our lightweight development server handled our target load easily — the connection count barely moved over the course of the test, which meant that most requests were processed as fast as they came in. This also means that we know what to do if we need to increase capacity — basically, scale the database so it allows more connections and processes queries faster.

The other question that stress testing solved was verifying behaviour under pressure. I ran a second test at 5 x {secret number} with a lot more I/O, in order to saturate the database connections.

We use Fastify for our server framework, and it has a @fastify/under-pressure plugin built just for this scenario. We're monitoring a few metrics, including the number of active database connections relative to the budget we've given the server module, and made it so that the server starts rejecting requests with 503 Service Unavailable and a Retry-After header specifying the delay after which the client can try again. This creates the back pressure that tells clients to back off, and lets the connection pool recover.

Running the saturation test demonstrated that the back pressure worked as intended: when the connection count hit 85% of the limit, the server started to reject requests. Nothing crashed, memory use didn't spike, CPU wasn't saturated, and when the clients backed off, everything returned to normal. I tried using Tarinoi while the saturation test was ongoing, and the only visible indication of it was that the connectivity icon went from "connected" to "backoff" for a while; when the pressure abated, the client reconnected and re-synced. Since Tarinoi is offline-first, I could continue to work as normal. I would probably not even have noticed unless I happened to have an active co-op editing session going – that would have been interrupted when the server saturated.

After this, I'm confident that Tarinoi can easily handle {secret target number} of active users we're hoping to get -- and that it'll stay up and recover gracefully even with "thundering herd" type incidents where usage suddenly spikes far beyond the usual limit. I also know what we'll need to do when we want to add capacity, and how to verify that the added resources had the impact we expected.