How we protect our most sensitive secrets from the most determined attackers

Read the article

As a bank, we have a lot of secrets we need to protect to keep our customers and ourselves safe. These include private keys that only we know, which we use to digitally sign requests as coming from us (sort of like a private password). These private keys are super important and are used all over Monzo and when interacting with payment networks (such as Faster Payments) to prove that a request genuinely comes from Monzo. If someone got hold of these private keys, they could impersonate us when talking to other financial institutions. We don’t want this to happen.

The way we prove that requests are authorised by us is through a system called Public Key Infrastructure (PKI for short). In PKI, everyone involved is issued a Digital Certificate that states:

  • Who they are

  • What they’re allowed to do

  • Who issued the certificate

Because every certificate is issued by someone else, there has to be a final certificate at the top of the system that everyone trusts and which starts the issuing process for all certificates below them. This is called the Root Certificate and it's managed by a Root Certificate Authority. This Root Certificate is super important, because it can issue certificates to do basically anything which could cause massive damage if misused.

I want to take you through our Root Certificates, who can manage them, how they are protected and how these layered controls work together to make sure nobody can access the certificate without permission.

Why do we need this?

When computers talk, they need to be able to check that the person they're talking to is who they say they are. Your own computer is doing this all the time; when you connect to monzo.com your computer is checking that the server you're connecting to is actually monzo.com by validating the certificate. For example, it’s checking that the certificate is actually issued to monzo.com and not another website.

One of the many steps to making sure a Digital Certificate is valid is checking whether it has been issued by a valid Certificate Authority. These are organizations that your computer trusts to issue Digital Certificates, which have been installed by the manufacturer of your computer and operating system.

You can look at these on your computer if you’d like. Here’s how:

  • On Windows: Press the Windows key + R, type in “certmgr.msc”, press enter, click on Trusted Root Certification Authorities, then click Certificates

  • On Mac OS X: Press ⌘ and space, type in “Keychain Access”, press enter, click on System Roots

  • On Linux: You probably don’t need verbose instructions 😛 run “ls /etc/ssl/certs”

A Digital Certificate issued by LetsEncrypt R3 (a Certificate Authority) to community.monzo.com

A Digital Certificate issued by LetsEncrypt R3 (a Certificate Authority) to community.monzo.com

These are the certificates your computer trusts, but inside companies, payment networks and banks we trust additional certificate authorities, because we have different needs. For example:

  • the Faster Payments network has its own certificate authorities, which are used to check a transaction actually comes from the bank it says it does

  • when exchanging financial crime data with the National Crime Agency (NCA) we use private certificate authorities to confirm we’re talking to the NCA, and the NCA confirms they’re talking to us

Public certificate authorities are pretty secure, but they’re only really built to verify who controls a web domain (this is called Domain Validation). We need to use certificates for more than what public certificate authorities can support (Extended Validation exists, but it’s a broken standard that has been phased out for a reason)

We therefore needed to build our own Root Certificate Authorities to supplement our needs.

What is the threat against us?

Like all UK banks, we need to be prepared as we may be targeted by very determined attackers (we call these ‘threat actors’). So our security has to be top notch.

These threats can come from many places:

  • The supply chain: someone may modify an item before its shipped to us or while it's in transit

  • Externally: An attacker may attempt to commit industrial espionage / another attack against us, there’s a few things this could involve, like breaking into our office, coercing an employee into giving up information (willingly or unwillingly) or stealing sensitive equipment from us.

  • Internally: An employee may attack us from the inside, abusing their position to access private material.

COTTONMOUTH-1, a USB cable manufactured by the NSA that looks like a normal USB cable, but has a hidden implant that allows an attacker to inspect data on the cable wirelessly, modify data as it travels over the cable and install malicious software on connected computers

- US National Security Agency, Advanced Network Technology (ANT) Division

COTTONMOUTH-1, a USB cable manufactured by the NSA that looks like a normal USB cable, but has a hidden implant that allows an attacker to inspect data on the cable wirelessly, modify data as it travels over the cable and install malicious software on connected computers - US National Security Agency, Advanced Network Technology (ANT) Division

We don’t have many assets that are valuable just by virtue of what they are. We don’t have a big vault full of cash, or precious metals. Our private keys are valuable, but only because of what they’re used for.

This means that we don’t need to worry as much about detected unauthorized access. Why? If we detect an attack while it is in progress, we can move to lock down our systems and stop the attack. Since getting these keys requires physical access, we can respond to a detected attack with physical force. If we detect an attack on our physical security or unauthorized access, we can take immediate steps to disable the keys and stop them from being useful.

If we were holding cash or precious metals, handling unauthorized access would be much more of a problem, because you can’t remotely invalidate bundles of cash.

This means we needed to build a system where we:

  • Could securely generate and manage private keys, and use them to sign things when authorized

  • Cannot lose access to the keys by accident

  • Could demonstrate that the system components involved haven’t been tampered with

  • Don’t rely on a single employee, ideally we don’t rely on a group of employees

  • Have high auditability, being able to prove to Monzo employees (and other folks) that the system has not been compromised

  • Have little to no opportunities to compromise the system, at least not in a way that wouldn’t be detected

What does our system look like?

Here’s a high-level diagram of our system:

Architecture Diagram showcasing how the various components such as the Air Gapped Laptop, Hardware Security Modules, Operating System and Keyholder Smart Cards come together

Let’s talk through the individual components.

The Air-Gapped Laptop

Our entire system is air-gapped, which means it is physically isolated from the outside world and has no way to connect to the internet. This reduces the attack surface significantly, as you cannot break into the system from a remote computer over the internet, instead you have to do it physically. This is an important security feature and it means that the laptop cannot talk to anyone else. 

We’ve achieved this by disabling a lot of the laptop’s features, for example it cannot connect to any wireless network, so we don’t have to worry about it doing so maliciously. We’ve also taken other measures to frustrate attackers, for example we have physically removed the hard drive so there is no way to persist data on the laptop itself

Operating System (OS) + Material CD-R

The laptop needs an Operating System (OS) to boot. The Operating System is the program that runs on your computer to provide your desktop environment (such as Windows or Mac OSX). Because the Laptop has no storage of any kind, it has nowhere to store an operating system. I refer to it as “a blank canvas”. To boot an operating system we have to supply it ourselves.

We have a series of CD-R disks (just like the ones you may have at home for very old music) that contain the operating system we use, which is a modified version of an OS called the Ceremony Operating ENvironment / coen. All of the changes we have made have been peer reviewed and audited by multiple security engineers.

The source code for our version of coen is open for anyone at Monzo to read. The process for building this Operating System guarantees that you will get exactly the same file every single time for the same source code. This means anyone with access to the source code can check that we're using the same OS without any modifications by running the build commands from their own computers and verifying the output they get exactly matches our output.

We store the OS on CD-R disks because CD-R disks can only be written to once, so once the OS has been written to the CD, nobody can edit the OS. As an additional precaution, each CD is sealed inside a tamper-evident bag.

These CDs are burned in their own key ceremonies on camera with multiple witnesses. The CDs are then placed in a CD case, that case is placed into a tamper-evident bag, the bag is sealed, the unique bag number recorded and someone writes the details of what the CD is for onto the bag. Witnesses then sign the bag.

Sometimes we need to add other files to the laptop in order to perform tasks, such as bringing a request to sign a digital certificate (a Certificate Signing Request) onto the laptop so that we can sign it. For these we use a second CD-R, burned with the same precautions as the Operating System CD and stored in the same manner.

Hardware Security Module

Our Laptop doesn’t have possession of the private keys and never sees the keys in a raw form, instead the actual keys are created and used by our Hardware Security Modules. These are physical computing devices that help us manage our keys and keep them safe. They have their own physical safeguards so we know if they have been tampered with. The Hardware Security Module is the only place that actually holds a copy of the private keys and is where all cryptographic operations occur.

Keyholder smart cards and passwords

Our keyholders are the folks allowed to unlock our Hardware Security Module to access our private keys. We have between 6 and 12 Monzonauts who are keyholders, a certain number of them are necessary to unlock the Hardware Security Module (this is known as the quorum, I’m going to keep the exact number secret 🙊).

Each keyholder has a smart card which has been sealed in its own tamper evident bag. These tamper evident bags are only opened during key ceremonies to ensure that keyholders are only using them when authorised. When we wish to unlock the Hardware Security Module (for example: to sign a new Digital Certificate with one of our private keys), we insert the smart card of each keyholder into the Hardware Security Module. To unlock the smart card and prove it's really them, the keyholder then has to enter a password that only they know.

To make sure only the keyholder knows this password we have a few precautions:

  • We use a specific keyboard, which is stored in a safe and has its own chain of custody to make tampering with it difficult

  • We place an enclosure around the keyboard so that only the keyholder can see what they are typing

Once enough keyholders have inserted their smart cards and entered their passwords, the Hardware Security Module unlocks and allows us to use the private key material.

What’s the procedure for accessing this material?

Accessing this material is not routine, so when it is accessed we take lots of safeguards.

When accessing our secret material we record the entire process, we have at least 3 cameras recording the room, each camera is recording to a local hard disk as well as streaming to a remote server, so that if either recording fails there is a back-up. Cameras further overlap in the areas of the room they cover so that if a camera fails entirely, we still have recordings from the other cameras.

Our equipment and sensitive material is continuously recorded from the moment it leaves a tamper evident bag to the moment it enters a new one, tamper evident bags are explicitly recorded to show there is no evidence of tampering before being opened, as well as being inspected by the Ceremony Administrator.

We produce a script in advance detailing every action we’re going to take to access the material, here’s a redacted example of part of one of our scripts.

When material is accessed, at least the following people are in the room:

  • A Ceremony Administrator, who leads the ceremony

  • An Internal Witness, who audits the actions of the ceremony administrator

  • The Keyholders, who watch the Ceremony Administrator and Internal Witness, as well as unlock the HSM when asked to

Sourcing Equipment

Sometimes we have to purchase new equipment as part of maintaining this service. We have two methods for sourcing new items.

For equipment that is hard to tamper with or where tampering wouldn’t get you very far, we place regular orders from suppliers to ship to our office, but we don’t disclose they’re for the security team so that these orders blend in with our normal equipment orders. This equipment is then moved into a safe.

For equipment that is easier to tamper with or where tampering could become a larger problem, we make unannounced visits at random to retail stores and purchase off-the-shelf products. This makes sure an attacker can’t predict which items we’re going to purchase and so can’t tamper with them.

We purchase these items with a minimum of 3 employees present and watching. We immediately seal the equipment in a tamper evident bag. The item is never out of sight of all employees present until it is placed in a tamper evident bag and that bag is sealed. All employees physically present then sign the bag and individually make a note of the bag number. The bag is immediately returned to a safe.

Conclusion

Some of the secret material we have to store is very important, and we take our responsibility to keep this material safe very seriously. I hope this post has given you some insight into the measures we’re taking to keep some of our most sensitive secrets safe from unauthorised access.

We’ll publish a second blog post soon talking about the more technical details and decisions we made as part of this program, Watch this space 🔜

If you have any questions, feel free to drop by the forum thread for this post. My username is glcy and I’ll be happy to answer questions 😊