Rebooting Linux with encrypted disk
How to make it easier to keep the kernel updated
As an Ethereum solo staker, I want my Linux staking box to be as secure as feasible. It applies to physical security too. If the attacker gains access to the validator key stored on disk, it will allow him/her to do malicious things such as intentionally exposing the validator to the slashing penalties.
My machine isn’t placed in a data center guarded 24/7, therefore to protect it from the physical attacker I used cryptography.
Encryption inconvenience
I decided to use LUKS disk encryption which can be enabled easily during the Linux installation. I set a long passphrase and I rest assured my data is well protected.
I need to provide the passphrase each time I turn the machine on. This way the volume key used to unlock the disk, can be derived from the text I enter. It requires a keyboard and physical presence during startup, therefore it may be inconvenient for headless systems.
There are many methods to make it easier. Though I use one of them, I find them either complex, costly, unreliable, or compromising security. I don’t want to discuss them here, because this topic deserves a separate post.
The important thing is, once started, the node should stay online. And with UPS protection, turning it off doesn’t happen almost at all.
Maintenance
To ensure there are no security bugs I perform regular software updates. Modern Linux distributions make it really easy to automate.
However, kernel updates are not that simple. To use a new kernel, the system needs to be rebooted. It’s just a short downtime which is often acceptable.
But when using disk encryption, the disk needs to be unlocked. How to achieve that when using ssh connection from a remote location? Given the fact kernel updates can be frequent, it becomes an issue.
One way to approach it is to try to avoid reboots.
Live patching…
…is a technique to apply changes to the running kernel, without a reboot. There are many proprietary implementations provided as services. Preparing a binary patch ready to be applied to a running kernel seems to me a complex task that requires a lot of testing. Probably that’s why most suppliers charge for this service. In the case of Ubuntu Livepatch, before the patch is pushed to the customers, it is tested on the production systems of non-paying users. That’s the cost of using it for free!
But still, when your kernel gets a live patch it is recommended to reboot in the nearest maintenance window.
I like to use open-source solutions instead of proprietary ones. Therefore I needed something else.
No rebooting at all?
In some environments and threat models, most vulnerabilities in the kernel are not a big deal. If you trust your local users, expose one or a few regularly updated services, and firewall the rest of the ports, then it’s just unlikely for the attacker to succeed. He/she will need a remote exploit in the tried and tested TCP/IP stack. I believe those types of bugs are extremely rare.
In the above scenario, one may choose to keep running with the same kernel, avoiding reboots. For the record: I don’t recommend this approach.
That said, if the only reason for ignoring established good security practices is to prevent reboots because disk encryption makes it inconvenient, then I think we should do better.
We should make reboots convenient.
How I wish it would work
I would like to reboot to the new kernel without entering the passphrase again. It was entered once: can’t it be reused for a new kernel?
Well, we have kexec. It will be 18 this year, but it is not popular amongst people. If you haven’t heard about it, it allows you to replace the currently running kernel with a new one. And it does it without involving hardware reset and executing the boot loader:
$ sudo kexec -l new_kernel.img # load kernel into memory
$ sudo kexec -e # boot the loaded kernelWhile being cool, it’s not the same as live patching. The new kernel has to boot the entire system, filesystems need to be mounted and services have to start. The disk has to be unlocked, so the passphrase is required again.
What if it was possible to preserve the volume encryption key?
Linux keeps the keys for disk encryption in the kernel keyring. It is a special, secure place in memory. Even the root can’t retrieve volume encryption keys from there!1
$ sudo dmsetup table
dm_crypt-0: 0 36284416 crypt aes-xts-plain64 :64:logon:cryptsetup:edd435cc-b51a-4530-aef3-5010caeee165-d0 0 252:3 32768As you can see above, only key reference is provided. The actual key is kept securely in kernel memory.
If the user could tell kexec to keep the given key intact, then the new kernel could reuse it. The newly booted system could unlock the disk without asking for a passphrase. I imagine kexec adds a new option allowing to specify a key to preserve:
$ sudo kexec -l new_kernel.img --keep-key=my-key-referenceWe are not there yet
Most of the work related to the volume key preservation feature would need to be done on the kernel side, but kexec and cryptsetup would also need to implement support. From my limited perspective, it seems doable, but I cannot estimate the amount of work required.
In the meantime, let’s try to resolve this issue using the tools we already have.
Initramfs
Kexec allows to pass 2 things to the new kernel:
kernel command line containing various options for kernel and userspace,
initramfs which is run by a new kernel to initialize various things, including disk encryption.
We could include a volume key in the kernel command line and later use it to set up disk encryption. It would work, but the kernel command line is visible to the users, which disqualifies this concept.
So we are left with initramfs. This is a file located typically in an unencrypted /boot partition. We definitely don’t want to store the volume key on an unencrypted disk, because it defeats the purpose of encryption.
But we don’t need to modify the original initramfs file permanently. It’s enough to modify it temporarily and pass it to kexec.
Introducing cryptreboot
This led me to create a tool that:
Asks the user for a passphrase to derive the volume key.
Copies original initramfs into memory and patches it to include the volume key.
Uses kexec to load patched initramfs and kernel into memory.
Initiates standard system shutdown to stop services and unmount filesystems.
On shutdown completion, a kernel with patched initramfs is executed. This step is done by systemd.
Here is what it looks like compared to the standard reboot. If you prefer, you can check the identical YouTube version.
As you can see, the user is still asked for a passphrase, but it’s done before reboot, not after. This way the user could execute a reboot remotely. Physical presence during startup is not required anymore.
From a security standpoint, it should be almost as secure as the standard setup:
secrets are not persisted anywhere, everything is done in memory,
volume key touches userspace for just a brief moment; the rest of the time it stays safely in the kernel keyring.
If you have a different perspective, please leave a comment below.
Cryptreboot is an easy-to-use, drop-in solution written in Ruby and released under the MIT license. You can find it on GitHub.
What’s next
Here are the features I plan to implement:
boot loader configuration parsing; currently, cryptreboot relies on symlinks or the user to pick the kernel and initramfs,
optionally persist the volume key on an encrypted disk in a file accessible to root only; this way user accepting a slight security trade-off won’t be prompted for the passphrase at all,
integration with systemd, to make cryptreboot the default reboot handler,
deb package to make installation easier,
support for non-Debian Linux distributions.
If you use disk encryption in Linux, please give cryptreboot a try and share your feedback!
Thank you for reading this post. If you want to stay in the loop, subscribe and star the project on Github, so you won’t miss new cryptreboot releases.
It is actually possible to get the volume key from user space, but the disk has to be unlocked with a non-default —disable-keyring cryptsetup flag.