Security Engineering

I've been wanting to write something about our approach to security engineering with Tarinoi. It is crucial to any SaaS, and especially so in an industry where game content and stories are real business secrets that you really don't want to leak. The trouble is that it's a massive topic, entire textbooks have been written about it and there are people who make their whole careers about various aspects of it. It's also rather complicated and rather dull if you're not really into it.

With that preamble, here's that something – if you're not an engineer but care about keeping your stuff secure, I'm attempting to write this for you, without getting lost in the technical weeds, at a very general level.

Security engineering is all about defence in depth.

The Security Onion ​

A secure system is built like an onion, or if you prefer a more military metaphor, like a castle with a defence in depth consisting of multiple nested layers of fortifications. Since nothing on this mortal coil is perfect, we should assume that every layer has its weaknesses, but when we combine them, we'll get something that's stronger than any given part.

Some of the layers in Tarinoi's security onion are:

• Row-level security (RLS) at the database level. This means that every query to the database has to have the tenant ID (that's you, the customer, one game studio) explicitly set in the transaction in order to access tenant data. So a bug or a flaw in our SQL that would let a user inject a different tenant ID into the query, or omit it altogether, will return nothing.
• An object query language that's converted to SQL at a single point, with a required security pre-filter on every query, plus a hook for setting that tenant ID needed for RLS. We don't write bare SQL anywhere else, so that's just one module that we have to thoroughly test and only rarely update. This makes it much less likely that bugged SQL would ever even reach the layer where RLS kicks in.
• A client/server interface that uses a standard called JSON-RPC to pass requests and responses back and forth. This lets us use a single mechanism for validating requests and responses and enforcing permissions, and keeps the number of endpoints we have under control – fewer endpoints, less attack surface.
• A unified mechanism for defining access control policies and enforcing them for each request, and a JSON-RPC implementation that requires a defined access control policy. A single thing to test, always applied, can't be forgotten because the whole thing won't work if you do.
• An explicit access control policy called "internal" for traffic between Tarinoi components. It will reject any requests that don't include credentials only Tarinoi services have, in the unlikely event that somebody would leave an internal endpoint exposed to the public.
• A Kubernetes server that only exposes endpoints to the Internet if they're explicitly listed in a configuration (called an ingress), making it extremely unlikely that any traffic intended for internal endpoints will ever even reach them from the Internet.

There's a lot more, but you get the picture. Some of these layers are redundant, like the Kubernetes ingress and the internal access control mechanism, both of which block unauthorised access to internal server endpoints, or the application level security pre-filter and the database-level RLS, which both block cross-tenant data access. We build these redundancies in whenever we can anyway, because humans are flawed and somebody might screw up, causing one of the layers to fail.

Security Controls ​

When working on a software system, there are always more things to do than anyone is capable of doing. Things need to be prioritised. Usually, user-facing problems will go to the front of the queue, followed by new, valuable features. Security-related activities tend to fall down the priority list. This is bad for all kinds of reasons. Even if you do nothing at all, somebody will discover vulnerabilities in your dependencies, and you should patch those before anyone exploits them, even if the potential exploits are hidden under several layers of the security onion – as they usually are.

We've addressed this by building in as many automated security controls into our development process as we can. We have pre-commit hooks and DevOps pipelines that run automatically on every commit and in every merge request. Some of these are about plain old quality – automatically running our test suites and failing if they don't pass – but others are specifically security-related. We check every commit with a commonly used scan that looks for known bad security patterns, we run another one to check our deployment configuration for similar flaws, and we have a pipeline that runs on a schedule to look for vulnerabilities in our dependencies so we catch and patch them early.

Some of these controls protect against a particularly pernicious kind of security failure – lapses by people who know what they're doing but forget. These are surprisingly common and can leave security holes that are much bigger than an unpatched vulnerability.

For example, an experienced developer who definitely knows better might implement an entire server API but then just forget to wire up the already existing access control system. We have a measure in place to catch this: the JSON-RPC system we have doesn't work without a specified access control policy, and we have an allowlist for server requests that don't require authentication – such as those used for sign-up. If a developer disables security on and endpoint or makes a new one with no access control enforcement, it'll get caught in the pre-commit hook (or at the latest in the pipeline) because it's not on the allowlist – and putting it on the allowlist is a very deliberate action that will make them think twice.

The Simplest Policy That Could Possibly Work ​

Security has been a consideration in Tarinoi since before we wrote the first line of code. We designed the data structures so that we keep different data categories in different places, and devised an access control policy framework that is simple enough to reason about, but sufficiently fine-grained to keep things locked down.

This is a balancing act. The simpler your policies, the easier they are to reason about, use, and test, but if they're too simple, you won't be able to enforce meaningful policies. "Signed out – everything denied, signed in – everything allowed" is the simplest possible policy, but it won't cut it for most real-life purposes. We do need to be able to segregate data access by tenant, distinguish between read, write, and admin actions, and, for example, make it so that not everybody working on a project will be able to maintain payment information or upgrade the organisation account.

With Tarinoi, we landed with a matrix of nine squares: three data categories, three actions that can be performed on them. We have personal data, project data, and organisation account data, and we have read, write, and admin access. This means that every request that's sent to the server can be put in one of these boxes and validated against the permissions of the user making the request. So, for example, downloading a project to your workstation is a read request on project data, but updating your billing info is an admin request on organisation account data.

This thinking goes all the way down to the basic data structures. For example, we keep personal data in its own database tables and never mix it into project data, and where they do need to be connected, we use an opaque key that can only be resolved if the matching user data exists in the system. So, if you as a user delete your account, any information related to that account in project data becomes un-decipherable – the only thing that's left is an ID that doesn't point to anything. This also makes it easier to comply with data protection regulations – we don't need to dig out personally identifying information that falls under the GDPR, because it's already segregated into its own tidy structure that we can let you download, update, or delete to exercise your rights.

The Red Queen's Race ​

In Alice in Wonderland, the Red Queen was famously running as fast as she can just to stay in the same place. This is the case with security engineering. It's a continuous background workload and sometimes gets in the way of fixing bugs or making cool features. Sometimes we even have to nix something because allowing it would require data access that might be exploitable. It is worth the effort. Just like we don't ever want Tarinoi to be the bottleneck for the scale, complexity, and narrative richness of your game projects, we don't ever want it to be the weak link in your security chain. I've previously worked with some highly security-conscious customers, from finance, healthcare, government, even radiation security, so I am not new to this. We don't claim to be perfect, but we are running as fast as we can so you don't have to.