From VMs to Containers to Podman

Author: Sasank Chilamkurthy

Security means different things to different people. For us at JOHNAIC, it boils down to a simple operational principle: one user should not be able to interfere with another user unless they explicitly give permission. This sounds obvious, but it has deep implications for how we architect our systems.

The Sudo Problem

Linux has a special user called root who can modify anything on the system. Ordinary users can temporarily borrow root privileges using the sudo command. This is convenient for developers who need to install compilers, system libraries, or daemons.

But sudo is also dangerous. A malicious or careless user with sudo access can delete databases, crash the kernel, or spy on other users. From a multi-user server perspective, giving someone sudo is almost the same as giving them the keys to the building.

Yet a system without sudo is too restrictive for most developers. The question is how to give users the power they need without compromising everyone else.

Virtual Machines

The traditional answer is virtualization. A virtual machine runs a complete operating system inside another operating system. Users inside the VM can have root access, but their actions are isolated from the host. Even if they crash the VM, the host and other VMs remain unaffected.

This approach works, but it has overhead. Each VM needs its own kernel, its own memory reservation, and its own virtual devices. For AI workloads, virtualization is especially tricky because GPU drivers from Nvidia and others often restrict passthrough to enterprise customers.

Containers

Containers are a lighter alternative. Instead of virtualizing hardware, they share the host kernel and isolate only user-space processes. This makes them faster to start and more efficient to run. Containers have become the standard way to ship and deploy server software.

However, not all container engines are equally secure. Docker, the most popular engine historically, runs as a daemon with root privileges. When a user runs docker run ubuntu, they are actually asking the root-owned daemon to create a container. That container also runs as root by default. In effect, Docker permission is very close to root permission.

Podman and Rootless Containers

Podman is an alternative container engine developed by Red Hat. Unlike Docker, Podman has no daemon. When a user runs podman run ubuntu, the container is spawned directly by the user process. The container inherits the user's permissions, not root's. This is called rootless containerization.

With rootless Podman, one user cannot kill another user's containers or access their files. If a container is compromised, the attacker is limited to the user's own resources. This aligns perfectly with our operational definition of security.

On JOHNAIC, we use rootless Podman as the default container engine. Users get the developer experience they expect, but the system remains secure by default.

Published on 05/10/24