Kubernetes certs expiration forever solved

Overview

By default Kubernetes client certificates generated by kubeadm expire after 1 year. When that happens, you can no longer communicate with or control the cluster. Here’s how to resolve the issue permanently.

Prerequisite

Docker to use official image to build a customize kubeadm
kubectl to access cluster control plane
ssh access to cluster master node to upload and run customized kubeadm

Pre-Check

Before we start, let us log on the master node and get the certificates status.

1
kubeadm alpha certs check-expiration

We got 45 days left to renew our certificates.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Dec 02, 2022 23:24 UTC   45d             no
apiserver                  Dec 02, 2022 23:24 UTC   45d             no
apiserver-etcd-client      Dec 02, 2022 23:24 UTC   45d             no
apiserver-kubelet-client   Dec 02, 2022 23:24 UTC   45d             no
controller-manager.conf    Dec 02, 2022 23:24 UTC   45d             no
etcd-healthcheck-client    Dec 02, 2022 23:24 UTC   45d             no
etcd-peer                  Dec 02, 2022 23:24 UTC   45d             no
etcd-server                Dec 02, 2022 23:24 UTC   45d             no
front-proxy-client         Dec 02, 2022 23:24 UTC   45d             no
scheduler.conf             Dec 02, 2022 23:24 UTC   45d             no

And we have to make sure what kubernetes version exactly we are using.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
kubectl get node


# NAME    STATUS   ROLES    AGE      VERSION
# ip-xxxxx.compute.internal   Ready    master   2y320d   v1.16.1
# ip-zzzzz.compute.internal   Ready    <none>   2y249d   v1.16.1

kubeadm version


# kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T16:58:27Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

As we can see from both kubectl and kubeadm, v1.16.1 is the version we are going to us

Customize

Now we can build a customized kubeadm based on official upstream source code.

First of all, we need to fetch the original v1.16.1 source code from here https://github.com/kubernetes/kubernetes/tree/v1.16.1.

1
2
3
4
5
git clone https://github.com/kubernetes/kubernetes

cd kubernetes

git checkout -b my-v1.16.1 v1.16.1

Then we have two files to change:

./staging/src/k8s.io/client-go/util/cert/cert.go
./cmd/kubeadm/app/constants/constants.go

And the diff file below shows how we change them to for a 100-year certificates lifespan.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
diff --git a/cmd/kubeadm/app/constants/constants.go b/cmd/kubeadm/app/constants/constants.go
index 8cd287f31e3..18190647589 100644
--- a/cmd/kubeadm/app/constants/constants.go
+++ b/cmd/kubeadm/app/constants/constants.go
@@ -43,7 +43,7 @@ const (
 	TempDirForKubeadm = "tmp"

 	// CertificateValidity defines the validity for all the signed certificates generated by kubeadm
-	CertificateValidity = time.Hour * 24 * 365
+	CertificateValidity = time.Hour * 24 * 365 * 100

 	// CACertAndKeyBaseName defines certificate authority base name
 	CACertAndKeyBaseName = "ca"
diff --git a/staging/src/k8s.io/client-go/util/cert/cert.go b/staging/src/k8s.io/client-go/util/cert/cert.go
index 9fd097af5e3..865d6bb3a1a 100644
--- a/staging/src/k8s.io/client-go/util/cert/cert.go
+++ b/staging/src/k8s.io/client-go/util/cert/cert.go
@@ -63,7 +63,7 @@ func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, erro
 			Organization: cfg.Organization,
 		},
 		NotBefore:             now.UTC(),
-		NotAfter:              now.Add(duration365d * 10).UTC(),
+		NotAfter:              now.Add(duration365d * 100).UTC(),
 		KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,
 		BasicConstraintsValid: true,
 		IsCA:                  true,

Finally we can use Kubernetes official cross build image to build customized kubeadm. We can find the rigth version of image in the code repository.

1
2
3
cat ./build/build-image/cross/VERSION

# v1.12.10-1

Then we can pull this image from google registry:

1
docker pull gcr.io/google_containers/kube-cross:v1.12.10-1

We can lauch a container using this image to do build kubeadm binary in kubernetes source repository directory.

1
2
3
4
5
6
7
docker run --rm -v `pwd`:/go/src/k8s.io/kubernetes  \
    -it gcr.io/google_containers/kube-cross:v1.12.10-1  bash

# in container
cd /go/src/k8s.io/kubernetes

make all WHAT=cmd/kubeadm GOFLAGS=-v

Then we can find our customized kubeadm in _output/local/bin/linux/amd64/kubeadm. To check out our results we can run:

1
2
3
./_output/local/bin/linux/amd64/kubeadm version

# kubeadm version: &version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.1-dirty", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"dirty", BuildDate:"2022-10-18T20:44:33Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

Renew Certificates

It’s always a best practice to backup everything before we make any changes in production.

1
2
3
4
mkdir backups
cp /etc/kubernetes/pki/ backups/ -a
mv `which kubeadm` backups/ -a
cp /etc/kubernetes/kubelet.conf backups/

Then we have to replace kubeadm with our customized one.

1
cp /tmp/kubeadm /usr/bin/kubeadm

Finally we can perform the renew action and restart kubernetes control plane to effect.

1
2
3
4
5
6
7
8
9
kubeadm alpha certs renew all

# If you have multiple master nodes, run this command on
# every master node to restart all control plane containers
docker ps | grep -v pause | grep -E "etcd|scheduler|controller|apiserver" | \
	 awk '{print $1}' | awk '{print "docker","restart",$1}' | bash

# update client config if you
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

To checkout the results, we can run check-expiration again and we should see that certificates expiration won’t be a problem for us.

1
kubeadm alpha certs check-expiration

Summary

Do bear in mind that it is a not a best practice to use such long-term certificates which may cause security issues if certficates are not being kept properly. While in some environments on which you may be not willing to spend too much time to maintain or in most cases operate in a secured interal network, it can be a simple and easy way to do so.
kubeadm may differ if your Kubernetes version is not v1.16.x. Refer to official docs according to your own version.

Lei Zhilong

The best way to input is to output