Kubernetes certificate management on the Flow Enterprise cluster

Tags: Flow on-prem

Note: The commands in this article are available in Kubernetes version 1.19 and above. The format may change in future versions.

When installing Flow Enterprise Server, the underlying Kubernetes system generates Public Key Infrastructure (PKI) certificates for the internal use of the cluster, These certificates are used by components of the Kubernetes control plane and nodes to authenticate with each other.

Kubernetes manages these PKI certificates, but they are designed to expire after one year. Monitor the expiration dates of the cluster's PKI certificates and proactively update them once a year. If the certificates aren't updated, Flow will be unavailable and pods won't restart.

To see the current status of the certificates, run sudo kubeadm alpha certs check-expiration. Below is an example output indicating certificate expiration dates.

user@master.domain:$ sudo kubeadm alpha certs check-expiration

[check-expiration] Reading configuration from the cluster...

[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED

admin.conf                 Apr 05, 2022 22:09 UTC   363d                                    no      

apiserver                  Apr 05, 2022 22:09 UTC   363d            ca                      no      

apiserver-etcd-client      Apr 05, 2022 22:09 UTC   363d            etcd-ca                 no      

apiserver-kubelet-client   Apr 05, 2022 22:09 UTC   363d            ca                      no      

controller-manager.conf    Apr 05, 2022 22:09 UTC   363d                                    no      

etcd-healthcheck-client    Apr 05, 2022 22:09 UTC   363d            etcd-ca                 no      

etcd-peer                  Apr 05, 2022 22:09 UTC   363d            etcd-ca                 no      

etcd-server                Apr 05, 2022 22:09 UTC   363d            etcd-ca                 no      

front-proxy-client         Apr 05, 2022 22:09 UTC   363d            front-proxy-ca          no      

scheduler.conf             Apr 05, 2022 22:09 UTC   363d                                    no      


CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED

ca                      Mar 22, 2031 15:43 UTC   9y              no      

etcd-ca                 Mar 22, 2031 15:43 UTC   9y              no      

front-proxy-ca          Mar 22, 2031 15:43 UTC   9y              no 

Before updating your certificates, backup the /etc/kubernetes/pki directory of the primary node. Update the certificates anytime prior to expiration.

To update the certificates, run sudo kubeadm alpha certs renew all on the primary node.

Note: For any high-availability (HA) cluster where there are multiple primary nodes, run the command on all primary nodes.

user@master.domain:$ sudo kubeadm alpha certs renew all

[renew] Reading configuration from the cluster...

[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'


certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed

certificate for serving the Kubernetes API renewed

certificate the apiserver uses to access etcd renewed

certificate for the API server to connect to kubelet renewed

certificate embedded in the kubeconfig file for the controller manager to use renewed

certificate for liveness probes to healthcheck etcd renewed

certificate for etcd nodes to communicate with each other renewed

certificate for serving etcd renewed

certificate for the front proxy client renewed

certificate embedded in the kubeconfig file for the scheduler manager to use renewed

After running this command, the PKI certificates are updated on the cluster.

Next, update all users on the system that use the kubectl command. This updates their ~/kube/config file allows access to the cluster via the command line interface (CLI). If this isn't done, users will be unable to run kubectl to check the status of pods. To update the users, use the flow-enterprise-tools package and run sudo ./flow-kube-config -a from the flow-enterprise-tools/bin/ directory.

Next, either restart all Flow pods, or reboot the cluster and ensure that all pods restart without issue.

Then, check the new expiration dates for the PKI certificates to make sure they're updated.

user@master.domain:$ sudo kubeadm alpha certs check-expiration

[check-expiration] Reading configuration from the cluster...

[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'


CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED

admin.conf                 Apr 07, 2022 16:17 UTC   364d                                    no      

apiserver                  Apr 07, 2022 16:17 UTC   364d            ca                      no      

apiserver-etcd-client      Apr 07, 2022 16:17 UTC   364d            etcd-ca                 no      

apiserver-kubelet-client   Apr 07, 2022 16:17 UTC   364d            ca                      no      

controller-manager.conf    Apr 07, 2022 16:17 UTC   364d                                    no      

etcd-healthcheck-client    Apr 07, 2022 16:17 UTC   364d            etcd-ca                 no      

etcd-peer                  Apr 07, 2022 16:17 UTC   364d            etcd-ca                 no      

etcd-server                Apr 07, 2022 16:17 UTC   364d            etcd-ca                 no      

front-proxy-client         Apr 07, 2022 16:17 UTC   364d            front-proxy-ca          no      

scheduler.conf             Apr 07, 2022 16:17 UTC   364d                                    no     

Worker nodes do not require any certificate updates because the kubelet service is installed with auto-update when the kubelet client certificates are due for expiration.

back to top


If you need help, please email support@pluralsight.com for 24/7 assistance.