Kubernetes by Parts: etcd cluster (3)

etcd is a distributed key-value store. Kubernetes uses it for persisting all cluster data. (Not to be confused with application data, which is a different concept.) Kubernetes might as well be backed by a relational database such as PostgreSQL, or a key-value store such as Redis. For example, Rancher’s k3s defaults to sqlite3 instead. etcd is younger technology and specifically in the context of Kubernetes is more performant and reliable than the alternatives.

Prerequites 🔗︎

  • Peering certificates for etcd generated in the previous chapter. Either one certificate for all members, or three unique certificates.
  • The CA that issued (signed) those certificates.

Steps 🔗︎

In this part, we will deploy and etcd cluster directly onto the three linux servers you have prepared for the tutorial.

The following text will closely match official etcd docs: https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/clustering.md#tls Also consult https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/configuration.md (or the man page) for all the configuration flags you set.

First, install the etcd from a package (which tend to lag behind a little bit) or download the binaries from official release https://github.com/etcd-io/etcd/releases.

Skim through the options on etcd --help to get an idea of what is configurable. Take special interest those chapters:

Member — options for a single process that may differ between hosts. Obviously you want as few changes as possible to simplify deployment and operations, but there is nothing in etcd design that requires it.

Clustering — specifies how should the distinct etcd processes running on different hosts connect to form a cluster.

Security — somewhat complex but necessary configuration. The options follow the client cert and server TLS + CA for validation pattern we will see in all remaining Kubernetes components.

Prepare a set of three commands to start and etcd cluster on your servers. Use the certificates we generated in previous chapter for authentication and TLS encryption. Configure the processes to automatically form a three member initial cluster (as opposed to starting a process and manually adding members via etcdctl).

This is the final configuration for first etcd member we used:

start-etcd-1.bash
etcd --name etcd-1 \
  --listen-peer-urls https://0.0.0.0:2380 \
  --listen-client-urls https://0.0.0.0:2379 \
  --initial-advertise-peer-urls https://etcd-1.kbp.local:2380 \
  --advertise-client-urls https://etcd-1.kbp.local:2379 \
  --initial-cluster-token kbp-etcd-cluster \
  --initial-cluster etcd-1=https://etcd-1.kbp.local:2380,etcd-2=https://etcd-2.kbp.local:2380,etcd-3=https://etcd-3.kbp.local:2380 \
  --initial-cluster-state new \
  --client-cert-auth --trusted-ca-file=/opt/kbp/pki/ca.pem \
  --cert-file=/opt/kbp/pki/etcd-server.pem --key-file=/opt/kbp/pki/etcd-server-key.pem \
  --peer-client-cert-auth --peer-trusted-ca-file=/opt/kbp/pki/ca.pem \
  --peer-cert-file=/opt/kbp/pki/etcd-peer.pem --peer-key-file=/opt/kbp/pki/etcd-peer-key.pem \
  --log-level=debug
↪ Expand hint

Listen (bind) peer and client urls are set to 0.0.0.0 which means listen on all available interfaces. Initial advertisements only announce the local member. Cluster token is an arbitrary string identifying the cluster in a network. We use the initial cluster option to set all members before even starting the servers. Alternatively we could start a single etcd instance, add a second member manually, start the second etcd instance etc. With the initial cluster option we don’t have to rely on specific initialization order. Remaining options are the certificates we generated in a previous chapter.

If at this point a member fails to join the cluster with tls: failed to verify client’s certificate: x509: certificate specifies an incompatible key usage you generated its certificates with invalid CA profile and did not grant it both the server and client Key Usage. This is further discussed in the previous chapter.

Verification 🔗︎

Utilize etcdctl to connect to the newly formed cluster to list members. You will need to specify a CA for etcdctl to validate server certificates and both private and a public client key for the etcd cluster to authorize our access.

etcd member rlist

Querying the etcd cluster to verify all three pre-configured members successfully joined and have expected peering and client addresses. The screenshot is from etcd version 3.4 with new learner capabilities.

If not all three members are listed or you are refused connection altogether, investigate the logs. Common failures are invalid certificates (invalid signing profile, mismatched SANs, incorrect CA, …) and misconfigured cert paths.

Stabilization 🔗︎

The etcd processes we started are ephemeral and will not restart after a crash or system reboot. Create a systemd service to ensure etcd process re-launches.

/etc/systemd/system/etcd.service
[Unit]
Description=Kubernetes by Parts etcd
After=syslog.target
After=network.target

[Service]
ExecStart=/usr/local/bin/etcd --name etcd-1 \
  --listen-peer-urls https://0.0.0.0:2380 \
  --listen-client-urls https://0.0.0.0:2379 \
  --initial-advertise-peer-urls https://etcd-1.kbp.local:2380 \
  --advertise-client-urls https://etcd-1.kbp.local:2379 \
  --initial-cluster-token kbp-etcd-cluster \
  --initial-cluster etcd-1=https://etcd-1.kbp.local:2380,etcd-2=https://etcd-2.kbp.local:2380,etcd-3=https://etcd-3.kbp.local:2380 \
  --initial-cluster-state new \
  --client-cert-auth --trusted-ca-file=/opt/kbp/pki/ca.pem \
  --cert-file=/opt/kbp/pki/etcd-server.pem --key-file=/opt/kbp/pki/etcd-server-key.pem \
  --peer-client-cert-auth --peer-trusted-ca-file=/opt/kbp/pki/ca.pem \
  --peer-cert-file=/opt/kbp/pki/etcd-peer.pem --peer-key-file=/opt/kbp/pki/etcd-peer-key.pem \
  --log-level=debug

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
↪ Expand hint

Alternative steps and improvements 🔗︎

Instead of downloading binaries directly onto the servers and installing dependencies to the host, it is possible deploy etcd as a container on each host. One of the advantages is – as with containerization in general – somewhat simpler deployment. You will have to install a container runtime (which is otherwise needed only for the kubelet to connect to and run pods on). Note that nothing requires you to actually use the same container runtime kubelet will use. It makes sense from operational perspective, so we suggest planning ahead and using the same runtime for both.

Better yet, you can start the etcd containers as Kubernetes pods. Kubelet has a concept of static pods exactly for this purpose. Static pods have their complete specification directly on the host itself so they can start even without functional Kubernetes cluster. Traditionally Kubelet connects to kube-apiserver, which in turn connects to etcd.

The following timeline illustrates how the etcd cluster is formed with Kubelet:

  1. At first, no etcd process runs on any server (and thus kube-apiserver is not starting)
  2. Kubelet process is started, falls into a loop and does multiple things at once:
    • reads static pod configuration for etcd and creates the pod
    • reads static pod configuration for kube-apiserver and creates the pod
    • attempts to connect to kube-apiserver and fails
  3. kube-apiserver containers flap between running and failed as they cannot connect to etcd
  4. kubelet loops and periodically outputs an error informing that it failed to connect to kube-apiserver
  5. etcd processes start and form a quorum
  6. kube-apiserver stops flapping and starts exposing kubernetes API
  7. kubelet connects to the kube-apiserver

We will not use this form of self-hosting as it results in complex component dependencies. During the series we will repeatedly illustrate broken/missing functionality and then deploy additional components. Self-hosting requires many components at once and as such is harder to learn the main concepts on. We suggest completing the series as is first and then building a self-hosted cluster.

Chapters 🔗︎