During my master’s thesis, I worked closely with Kubernetes controllers and operators. It was a great opportunity to really get to grips with their design principles and architecture. In this post, I want to share these findings and see how they relate to our everyday software design.

Controllers

Kubernetes allows end-users to declaratively specify their desired state of the cluster via the Kubernetes API server. To reach that desired state, Kubernetes uses a set of controllers. Controllers run in the Kubernetes control plane and are agents that monitor the current state of a cluster and send requests to the Kubernetes API server to bring it closer to the desired state. The non-terminating loop that Kubernetes performs to achieve the desired state is called the reconciliation loop. The figure below shows the Kubernetes controller manager executing the reconciliation loop:

The manager calls the controllers’ reconciliation or reconcile functions in each iteration. These functions get their desired state by fetching a set of Kubernetes objects and bring the current state closer to that desired state by operating on another set of Kubernetes objects. A simple example is a replication controller fetching replica objects and creating pod objects as a result.

Design principles of controllers

Kubernetes manages to keep a complex cluster running smoothly by letting these independent controllers collaborate. It’s like a well-oiled machine with many moving parts. To really understand how Kubernetes achieves this, we need to take a closer look at the architecture of these controllers. I’ve identified the following key design principles:

• Single-responsibility principle: The design of Kubernetes in small control loops controlling a single kind of object makes it more robust and easier to reason about. For example, the deployment controller is responsible for creating, upgrading, downgrading, and scaling deployments without any disruption or downtime. If any errors occur in this process, only this controller needs to be inspected. In comparison, a centralised orchestration system quickly becomes fragile and challenging to update, especially in the presence of unexpected errors or state changes.
• Idempotence: A fundamental principle of controllers is idempotence. Controllers being idempotent implies that repeatedly calling the reconcile function with the same desired state should always lead to the same results. If a given object is already in the desired state, the controller should perform no actions. If a given object is not in the desired state, the controller should try to move the object to the desired state. Idempotency effectively allows Kubernetes to heal itself when undesired state changes happen:

  

• Stateless: Controllers are stateless. They contain all their state in objects they track and manage. Important state should not be kept in memory. If tracking state is required, controllers should store it in a Kubernetes object. Because controllers base all their changes upon the perceived state, the controllers are fault-tolerant. When a controller crashes, a new controller can immediately take over.
• Non-blocking: Another essential principle of controllers is that their reconcile function should be non-blocking. When a controller sends a request to create an object in an iteration of the reconciliation loop, it should not wait for the request to complete. Controllers should use a Kubernetes job instead if they want to perform a long-running task. This allows other controllers to react to the creation of objects and prevents resource starvation of the controller’s pod. This principle allows Kubernetes to manage its resources more efficiently and leads to a more responsive system.
• Assume transient errors: Controllers assume that arbitrary errors can self-resolve. Only in the case where the error is deemed non-retriable and non-recoverable does the controller crash. Due to the many controllers working together asynchronously, certain conditions are often not fulfilled for a controller to proceed, resulting in errors. Controllers often have to wait for the creation of objects or other actions to occur. For example, fetching a non-existent object results in the Kubernetes API returning an error. Additionally, because Kubernetes uses optimistic concurrency, writes to objects are expected to fail. In case of an error, the reconciliation loop should not crash nor wait for the resolution of the error. Instead, it should exit prematurely. This causes the controller manager to call the reconcile function at a later point in time to check if the error is resolved. Controllers use an exponential backoff mechanism by default, but additional mechanisms are possible, such as a fixed number of retries. Because controllers assume transient errors, Kubernetes is resilient to unexpected changes in the cluster.

These principles are not new by any means:

• The single-responsibility principle is one of the 5 SOLID principles.
• Idempotency is a well known trait of RESTful APIs, allowing requests to be retried.
• Statelessness is a commonly applied principle in web server design. It allows any request to be served by any web server, allowing you to easily horizontally scale.
• Event loops enable non-blocking I/O requests in languages such as Javascript and Python.
• Marc Brooker explains in the article Timeouts, retries, and backoff with jitter that a surprisingly large number of request errors can be overcome by assuming that the error is transient and simply retrying the call.

While most of these principles are considered best practice already, I find it fascinating to see them being applied in various ways within such a complex system. That’s all I want to share for now, some appreciation for the design that keeps your Kubernetes clusters running smoothly.