证书期限
使用kubeadm安装的集群，一般根证书CA的期限是十年，其余组件，包括etcd、apiserver、controller-manager的期限需要一年一续签。
查看证书是否到期的方法：
- 使用kubectl访问集群，报错kube-apiserver不可达
## kubectl get nodes
The connection to the server 192.168.126.100:6443 was refused - did you specify the right host or port?- 下一步，在master节点发现kubelet无法启动，报错证书过期
## systemctl restart kubelet.service
## journalctl -xefu kubelet
... ...
-- The start-up result is RESULT.
Apr 17 14:56:15 k8s-master.lab.example.com kubelet[7434]: I0417 14:56:15.109672 7434 server.go:446] "Kubelet version" kubeletVersion="v1.23.0"
Apr 17 14:56:15 k8s-master.lab.example.com kubelet[7434]: I0417 14:56:15.110553 7434 server.go:874] "Client rotation is on, will bootstrap in background"
Apr 17 14:56:15 k8s-master.lab.example.com kubelet[7434]: E0417 14:56:15.120625 7434 bootstrap.go:265] part of the existing bootstrap client certificate in /etc/kubernetes/kubelet.conf is expired: 2023-02-12 18:37:21 +0000 UTC
Apr 17 14:56:15 k8s-master.lab.example.com kubelet[7434]: E0417 14:56:15.120730 7434 server.go:302] "Failed to run kubelet" err="failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory"
Apr 17 14:56:15 k8s-master.lab.example.com systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 17 14:56:15 k8s-master.lab.example.com systemd[1]: kubelet.service: Failed with result 'exit-code'.以上由于kubelet 在2023-02-12后，证书过期，无法启动。
- 在master节点使用kubeadm查看管理的证书到期期限
## kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Apr 09, 2023 06:54 UTC <invalid> no
apiserver Apr 09, 2023 06:53 UTC <invalid> ca no
apiserver-etcd-client Apr 09, 2023 06:54 UTC <invalid> etcd-ca no
apiserver-kubelet-client Apr 09, 2023 06:53 UTC <invalid> ca no
controller-manager.conf Apr 09, 2023 06:54 UTC <invalid> no
etcd-healthcheck-client Apr 09, 2023 06:54 UTC <invalid> etcd-ca no
etcd-peer Apr 09, 2023 06:54 UTC <invalid> etcd-ca no
etcd-server Apr 09, 2023 06:54 UTC <invalid> etcd-ca no
front-proxy-client Apr 09, 2023 06:54 UTC <invalid> front-proxy-ca no
scheduler.conf Apr 09, 2023 06:54 UTC <invalid> no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Apr 06, 2032 06:53 UTC 8y no
etcd-ca Apr 06, 2032 06:54 UTC 8y no
front-proxy-ca Apr 06, 2032 06:54 UTC 8y no续签证书
- master节点续签由kubeadm管理的证书
## kubeadm certs renew all //续签组件证书，但不包含kubelet
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.- 再次启动kubelet
## systemctl restart kubelet此时发现kubelet仍然无法启动
- 在master节点，把新签发的管理员凭证/etc/kubernetes/admin.conf重新分发给客户端节。
## scp /etc/kubernetes/admin.conf student@client:~/.kube/config- 检查各成员节点kubelet是否开启证书轮换，正常kubelet证书自动轮换在Kubernetes v1.19成为stable。
## NODES='k8s-master k8s-node1 k8s-node2'
## for x in $NODES;do ssh $x 'hostname && sudo grep rotateCertificates /var/lib/kubelet/config.yaml';done
k8s-master.lab.example.com
rotateCertificates: true
k8s-node1.lab.example.com
rotateCertificates: true
k8s-node2.lab.example.com
rotateCertificates: true- 关闭节点ntp时间同步，把节点本地时钟调整为过期之前，让kubelet能够启动后自动轮换证书。
## NODES='k8s-master k8s-node1 k8s-node2'
## for x in $NODES;do ssh $x sudo timedatectl set-ntp false;done
## for x in $NODES;do ssh $x sudo timedatectl set-time 2023-01-01;done
## for x in $NODES;do ssh $x sudo systemctl restart kubelet.service;done
## kubectl get nodes // 重启各节点kubelet后，发现集群启动
NAME STATUS ROLES AGE VERSION
k8s-master.lab.example.com Ready control-plane,master 322d v1.23.0
k8s-node1.lab.example.com Ready <none> 322d v1.23.0
k8s-node2.lab.example.com Ready <none> 322d v1.23.0
// 校验各节点kubelet证书过期时间
## for x in $NODES;do ssh $x "hostname && sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -text -noout | grep -A 2 Validity";done
k8s-master.lab.example.com
Validity
Not Before: Dec 31 23:55:51 2022 GMT
Not After : Dec 31 23:55:51 2023 GMT
k8s-node1.lab.example.com
Validity
Not Before: Dec 31 23:55:51 2022 GMT
Not After : Dec 31 23:55:51 2023 GMT
k8s-node2.lab.example.com
Validity
Not Before: Dec 31 23:55:51 2022 GMT
Not After : Dec 31 23:55:51 2023 GMT- 开启各节点时间服务器自动同步
## for x in $NODES;do ssh $x sudo timedatectl set-ntp true;done
## for x in $NODES;do ssh $x sudo timedatectl;done
