Reconciler in kube-apiserver

When deploying a cluster manually, you may have noticed --endpoint-reconciler-type option on kube-apiserver.

The official documentation offers little help:

--endpoint-reconciler-type   string     Default: "lease"
Use an endpoint reconciler (master-count, lease, none)

--apiserver-count            int        Default: 1
The number of apiservers running in the cluster, must be a positive number.
(In use when --endpoint-reconciler-type=master-count is enabled.)

https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

Diving into the code, I’ve uncovered this comments:

DefaultEndpointReconcilerInterval is the default amount of time for how often the endpoints for the kubernetes Service are reconciled.

DefaultEndpointReconcilerTTL is the default TTL timeout for the storage layer. https://github.com/kubernetes/kubernetes/blob/master/pkg/master/master.go#L114-L118

Kubernetes service reconciler 🔗︎

One of many kube-apiserver tasks is updating endpoints for kubernetes service (often available as kubernetes.default.svc.cluster.local). Applications connect to this service when communicating with kube-apiserver in-cluster. Go implementation of Kubernetes API defaults to $KUBERNETES_SERVICE_HOST (https://github.com/kubernetes/client-go/blob/master/rest/config.go#L415), which contains IP of the service, which is then translated to one of the concrete apiserver IPs.

It may be surprising that kube-apiserver manages this service, as endpoints are updated by kube-controller-manager. https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/ The reason is pragmatic: kube-apiserver only relies on etcd and coupling it with additional components would limit reusability.

Reconciler type master-count 🔗︎

Master count is an older implementation that relies on all kube-apiservers being configured with --apiserver-count with identical value.

pkg/master/reconcilers/mastercount.go

ReconcileEndpoints sets the endpoints for the given apiserver service (ro or rw). ReconcileEndpoints expects that the endpoints objects it manages will all be managed only by ReconcileEndpoints; therefore, to understand this, you need only understand the requirements and the body of this function.

Requirements:

  • All apiservers MUST use the same ports for their {rw, ro} services.
  • All apiservers MUST use ReconcileEndpoints and only ReconcileEndpoints to manage the endpoints for their {rw, ro} services.
  • All apiservers MUST know and agree on the number of apiservers expected to be running (c.masterCount).
  • ReconcileEndpoints is called periodically from all apiservers.

The implementation appends the IP of kube-apiserver that is executing the function, sorts the list and removes subsequent IPs (wrapping back to the beginning of the list if necessary) until it contains at most apiserver-count elements. This makes it eventually consistent when the function is executed on all servers.

I believe this reconciler does not remove the endpoints when they become unavailable and once set, it only replaces them when a new one starts reporting in. If I’m mistaken please let me know!

Lease reconciler 🔗︎

This option is well commented in the source code (pkg/master/reconcilers/lease.go):

ReconcileEndpoints lists keys in a special etcd directory. Each key is expected to have a TTL of R+n, where R is the refresh interval at which this function is called, and n is some small value. If an apiserver goes down, it will fail to refresh its key’s TTL and the key will expire. ReconcileEndpoints will notice that the endpoints object is different from the directory listing, and update the endpoints object accordingly.

Unlike master-count this reconciler relies only on etcd, does not depend on configuration and can dynamically scale the number of kube-apiserver instances. If a kube-apiserver instance become unavailable and does not update its lease in etcd until the TTL, other runs of the reconciler will remove it from the list of kubernetes service endpoints.

Lease option seems to be better than master-count in every meaningful aspect.