Kubernetes by Parts: Certificate Management (2)

Kubernetes and the supporting components rely heavily on TLS for security. In this chapter, we will explain the architecture, create multiple certificate authorities and issue all server and client certificates.


Overview of certificate structure used in Kubernetes deployment. Circles represent certificates used for client authentication (KC, CC), server authentication (CS), or for both in case of etcd peering (P). All of the etcd certificates (P, CS, CC) are issued by etcd CA, which is represented by the topmost yellow rectangle. Remaining certs (KS, KC, LC and LS) will be issued and verified by a different CA which we will refer to as Kubernetes CA.
Overview of certificate structure used in Kubernetes deployment. Circles represent certificates used for client authentication (KC, CC), server authentication (CS), or for both in case of etcd peering (P). All of the etcd certificates (P, CS, CC) are issued by etcd CA, which is represented by the topmost yellow rectangle. Remaining certs (KS, KC, LC and LS) will be issued and verified by a different CA which we will refer to as Kubernetes CA.

The official documentation lists some requirements at PKI certificates and requirements. You should refer to this document when generating the certificates yourselves to match the expected subjects, kinds and SANs (hosts).

The above diagram is simple, yet it conveys a lot of information. For example, it shows that to start an etcd process, we will need:

  • Certificate for peering (P on the diagram) issued by etcd CA (the large topmost rectangle).
  • Certificate for the server-side of client connections (CS), also issued by etcd CA.
  • And also the public etcd CA itself to verify incoming requests.

Similarly, to start a kube-apiserver process, we will need:

  • Client certificate to authenticate requests to etcd. This effectively functions as a password/token that etcd will verify and eventually grant us access to the data. Since etcd process will be doing the trust-chain verification, this certificate must be issued by etcd CA (as visualized by the large topmost rectangle on the diagram).
  • Client certificate to authenticate requests to kubelet.
  • etcd CA to verify server certificates on the client-side. If kube-apiserver didn’t validate the server certificates, we would be open to a man-in-the-middle attack. It would be similar to using plain HTTP instead of HTTPS when visiting a website.
  • Server certificate issued by Kubernetes CA for the secure API that we want kube-apiserver to expose (the one that defaults to port 6443). This cert is used for TLS and is what makes the API available on HTTPS.
  • Kubernetes CA to verify incoming requests to the kube-apiserver. This is one of the many authentication methods kube-apiserver provides (such as OIDC and HTTP Authorization).

Both etcd and kube-apiserver authorize incoming requests by validating the trust chain. If we were to use the same CA, certificates issued for etcd could be misused for kube-apiserver and vice versa. The following diagram illustrates why using a single CA is insecure.


Illustration of security implications of using a single CA for both etcd and for kube-apiserver.
Illustration of security implications of using a single CA for both etcd and for kube-apiserver.

The KC certificate which was originally issued for kubelet was intended for authentication to the kube-apiserver and has client key usage. When the same CA validates certificates for both kube-apiserver and for etcd, there is literally nothing preventing the this KC certificate from being misused for direct communication with the etcd cluster. This bypasses all authorization mechanisms built in to kube-apiserver and grants unlimited privileges to whomever gets hold of this KC certificate.

Later on in this series, we will utilize TLS bootstrapping, so we will also separate CAs for kube-apiserver server itself and client authorization, as we will need to upload a private key to a remote controller so it can dynamically issue kubelet certificates. This will be explained in a later chapter.

Issuing a Certificate Authority 🔗︎

Let’s start by generating etcd CA. I suggest cfssl https://github.com/cloudflare/cfssl because the configuration is simple and does not rely on external files, unlike openssl, but any option is fine as long as you can generate the certificate.

Ideally we would limit CAs to a domain, but cfssl does not support name constraints (issue #713). For the purposes of this tutorial, we will use cfssl instead of openssl for better readability.

Use cfssl to generate a certificate which we will use as a CA for etcd:

This series is supposed to be followed slowly over multiple hours or days. We will hide all snippets by default. Take a peek to understand the syntax or copy the solution if you are stuck.

etcd-ca-csr.json
{
    "CN": "Kubernetes by Parts: etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "ca": {
        "expiry": "8760h"
    },
    "names": [
        {
            "L": "Kubernetes by Parts",
            "O": "Clusterise",
            "ST": "etcd"
        }
    ]
}
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca -
↪ Expand hint

The expiration is explicitly limited to 365 days; the default expiry is 5 years. cfssl unfortunately does not support units larger than an hour and the readability takes quite a hit. The object in names are set to arbitrary values that look good when inspecting the certificate (see RFC 5280 4.1.2.4 for all available options).

Certificates contain what they can be used for (X509v3 Key Usage). The CA we just generated has extensions that allow it to sign additional certificates. If those extensions were not set, the “CA” could still mechanically-speaking sign other certificates, but the trust chain would be invalid.

We will be generating quite a number of certificates, as illustrated in the overview diagram above. It’s preferable to limit the certificates to the least amount of usages they require. To help us with that, we will prepare a CA config with expirations and key usages. Consult the CoreOS documentation for signing profiles.

ca-config.json
{
    "signing": {
        "default": {
            "expiry": "8760h"
        },
        "profiles": {
            "server": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ],
                "expiry": "8760h"
            },
            "client": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ],
                "expiry": "8760h"
            },
            "peer": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ],
                "expiry": "8760h"
            }
        }
    }
}
↪ Expand hint

The expiry needs to be explicitly set for all profiles, the default is not applied when a profile is selected. All we have to do now is create CSRs (certificate signing requests) for all of the components and let the CA generate the actual keys.

Issuing etcd certificates 🔗︎

Our overall mission is deploying a Kubernetes cluster, for which we need etcd. etcd in turn needs two kinds of TLS certificates: peering for member communication inside the cluster and client for kube-apiserver to send requests to etcd.

We only need the peering certs to form an etcd cluster, so let’s start with that. Create a CSR for etcd peering+server with appropriate SANs (hosts).

etcd-shared-server-csr.json
{
    "CN": "Kubernetes by Parts: shared etcd peering and servers",
    "hosts": [
        "127.0.0.1",
        "etcd-1.kbp.local",
        "etcd-2.kbp.local",
        "etcd-3.kbp.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "L": "Kubernetes by Parts",
            "O": "Clusterise",
            "ST": "etcd",
            "OU": "etcd peering and server"
        }
    ]
}
cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=peer etcd-shared-server-csr.json | cfssljson -bare etcd-shared-server
↪ Expand hint

To keep things simple, the certificate we generated is valid for all three etcd members. The etcd-1.kbp.local etcd etc are routable from all servers. If the DNS was not setup properly IPs could have been used as well. etcd documentation state the loopback (127.0.0.1) is mandatory.

When generating the certificate for etcd member peering, we used the peering CA profile (prescient naming scheme). This grants Server Authentication key usage, which is checked when a member is attempting to join a cluster and also the Client Auth key usage.

With this certificate, we are ready to bootstrap an etcd cluster.

Even though etcd allows specifying separate certificates for peering and for the server-side of client (=out of cluster) communication, the server-side cert needs certificate with both the server and the client Key Usage. Issuing a certificate without client usage fails with tls: failed to verify client’s certificate: x509: certificate specifies an incompatible key usage. At that point the server can be given the same certificates as the peering. The server should not need client usage https://github.com/etcd-io/etcd/issues/9398. However, etcd reuses the server certificate as a gRPC gateway client https://github.com/etcd-io/etcd/issues/9785#issuecomment-396715692.

Supporting materials 🔗︎

Alternative steps 🔗︎

  • The root authority could issue a intermediate authority for each component. We would still need to distribute the intermediates and configure the components to verify clients using those certs.
  • It is recommended to create and sign a new key pair for every member in a cluster. We simplified by generating a single cert with SANs of all etcd hosts. See https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/security.md
  • Issue the CAs with name constraints. This requires using openssl as cfssl does not support constraints at the time of writing.

Chapters 🔗︎