Canonical Livepatch is a security patching automation tool which supports reboot-less security updates for the Linux kernel, and has been architected to balance security with operational convenience. Livepatch remediates high and critical common vulnerabilities and exposures (CVEs) with in-memory patches, until the next package upgrade and reboot window. System administrators rely on Livepatch to secure mission-critical Ubuntu servers where security is of paramount importance.
Since the Linux kernel is an integral component of a running system, a fault would bring the entire machine to a halt. Two complementary security implementations provide safeguards against malicious code from being inserted via Canonical’s live kernel patching functionality:
- Secure Boot ensures you’re running a trusted kernel
- Module signature verification ensures only trusted code is loaded into the kernel at runtime
Secure Boot ensures trustworthiness of binaries by validating signatures, they must be signed by a trusted source. It protects the Ubuntu machine by preventing user-space programs from installing untrusted bootloaders and binaries. Secure Boot validation results in a hard requirement for module signature verification, to insert code at runtime.
Livepatching the Linux kernel securely
There are multiple layers of protection ensuring Livepatch runs safely:
Firstly, the Livepatch Client is packaged and distributed as a self-updating snap application. Snap packages are tamper-proof, GPG-signed, compressed, and read-only filesystems. The self-updating functionality is clever enough to roll back to the previous version, if the upgrade fails. Snaps run in a sandboxed environment, and system access is denied by default. The Livepatch snap application is strictly confined, and has granular access only to the areas of the system that are essential for its function, through pre-defined snap interfaces.
Secondly, Canonical has implemented a certificate-based trust model to ensure Livepatch updates have been published by a trusted source, and not a third party with nefarious intent.
Certificate-based trust model for runtime code insertion
Livepatch implements a certificate-based trust chain wherein all patches must be cryptographically signed by Canonical. Certificates are embedded in all Linux kernels built by Canonical, and Livepatch updates are verified against these embedded certificates before being applied at runtime. Additionally, CA certificates are stored in bootloader packages to validate kernel signatures during the Secure Boot process, but this is a separate validation system from Livepatch module verification.
In order for this system to work over time, two certificates require periodic renewal. Client authentication certificates must be updated to successfully access content from Canonical’s servers, and the certificate in Livepatch Client must match the module signing certificates embedded in the kernels. Launchpad plays a crucial role in the development, packaging, and maintenance of Ubuntu. Launchpad’s build farm compiles source code into .deb packages, and hosts the CI/CD processes around maintaining a valid certificate for Livepatch.
The Livepatch engineering team and kernel engineering team collaborate with each other to ensure the kernels and Livepatch Client are using the appropriate certificate, and collaborate with the Launchpad team to ensure the builds have been signed appropriately. The Kernel Engineers at Canonical package the updates distributed by the Livepatch Client. The same machinery that is used for testing and validating the official kernel builds is repurposed for testing and validating every Livepatch update. Every Livepatch update is distributed as a signed kernel module, and the kernel validates module signatures against embedded certificates before applying the patch.
The public and private certificate pair must match to ensure the kernel can continue receiving Livepatch updates. Canonical signs every kernel with a private certificate, and the corresponding public certificate is embedded in the kernel at build time. All kernel modules, including the patches distributed by Livepatch, are signed with the appropriate private key. When Livepatch applies updates, both the Livepatch Client and the kernel validate signatures using the embedded public certificate. Mismatches between the public and private certificates embedded in the kernel for module signing validation will prevent Livepatch modules from being applied. Invalid Livepatch updates are simply rejected by the kernel during runtime signature verification.
Conclusion
The chain of trust established through Secure Boot, which ultimately requires signed kernel modules, ensures bad actors cannot use Livepatch as a vector for attack. Certificate expiry maintains the integrity of the trust chain, and ensures continued authorization to receive patches. For critical and high kernel vulnerabilities, organizations of all sizes and personal users alike, are turning to Livepatch to shrink the exploit-window of their Ubuntu instances, after kernel vulnerabilities are reported.