目录
一.系统环境
本文主要基于Kubernetes1.21.0和Linux操作系统CentOS7.4。
| 服务器版本 | docker软件版本 | Kubernetes(k8s)集群版本 | Kubernetes(k8s)集群升级后的版本 | CPU架构 |
|---|---|---|---|---|
| CentOS Linux release 7.4.1708 (Core) | Docker version 20.10.12 | v1.21.0 | v1.21.9 | x86_64 |
Kubernetes集群架构：k8scloude1作为master节点，k8scloude2，k8scloude3作为worker节点。
| 服务器 | 操作系统版本 | CPU架构 | 进程 | 功能描述 |
|---|---|---|---|---|
| k8scloude1/192.168.110.130 | CentOS Linux release 7.4.1708 (Core) | x86_64 | docker，kube-apiserver，etcd，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calico | k8s master节点 |
| k8scloude2/192.168.110.129 | CentOS Linux release 7.4.1708 (Core) | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
| k8scloude3/192.168.110.128 | CentOS Linux release 7.4.1708 (Core) | x86_64 | docker，kubelet，kube-proxy，calico | k8s worker节点 |
二.前言
在容器化应用的管理中，Kubernetes 是一个广泛使用的开源平台。为了保持集群的稳定性和安全性，定期升级 Kubernetes 是非常重要的。本篇博客将介绍如何在 CentOS 7.4 上升级 Kubernetes(k8s) 集群。
在 CentOS 7.4 上升级 Kubernetes(k8s) 集群的前提是已经有一套可以正常运行的Kubernetes集群，关于Kubernetes(k8s)集群的安装部署，可以查看博客《Centos7 安装部署Kubernetes(k8s)集群》https://www.cnblogs.com/renshengdezheli/p/16686769.html。
三.Kubernetes(k8s) 集群升级简介
Kubernetes 使用 kubeadm 工具来管理集群组件的升级。在集群节点层面，升级 Kubernetes(k8s)集群的过程可以分为以下几个步骤：
- 检查当前环境和配置是否满足升级要求。
- 升级master主节点（如果是多master，则master轮着升级）。
- 升级worker工作节点。
- 升级网络插件。
在软件层面，升级 Kubernetes(k8s)集群的过程可以分为以下几个步骤：
- 升级kubeadm
- 节点执行drain操作
- 升级各个组件
- 取消drain操作
- 升级kubelet和kubectl
注意：Kubernetes(k8s)集群升级的时候是不能跨版本升级的，比如：Kubernetes(k8s)集群 1.19.x可以升级为1.20.y，但是
Kubernetes(k8s)集群 1.19.x不能直接升级为 1.21.y，你只能从一个次版本升级到下一个次版本，或者在次版本相同时升级补丁版本。 也
就是说，升级时不可以跳过次版本。 例如，你只能从 1.y 升级到 1.y+1，而不能从 1.y 升级到 1.y+2。
四.升级master主节点
4.1 升级kubeadm
Kubernetes(k8s)集群版本是v1.21.0。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span>NAME STATUS ROLES AGE VERSIONk8scloude1 Ready control-plane,master 83d v1.21.0k8scloude2 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span> NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.0 k8scloude2 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0 k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0[root@k8scloude1 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.0 k8scloude2 Ready <none> 83d v1.21.0 k8scloude3 Ready <none> 83d v1.21.0
查看可用的kubeadm版本。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># yum list --showduplicates kubeadm --disableexcludes=kubernetes</span>已加载插件：fastestmirrorRepodata is over <span class="token number">2</span> weeks old. Install yum-cron? Or run: yum makecache fastDetermining fastest mirrors已安装的软件包kubeadm.x86_64 <span class="token number">1.21</span>.0-0 @kubernetes可安装的软件包kubeadm.x86_64 <span class="token number">1.6</span>.0-0 kubernetes<span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span>kubeadm.x86_64 <span class="token number">1.21</span>.0-0 kuberneteskubeadm.x86_64 <span class="token number">1.21</span>.1-0 kuberneteskubeadm.x86_64 <span class="token number">1.21</span>.2-0 kuberneteskubeadm.x86_64 <span class="token number">1.21</span>.3-0 kuberneteskubeadm.x86_64 <span class="token number">1.21</span>.4-0 kuberneteskubeadm.x86_64 <span class="token number">1.21</span>.5-0 kuberneteskubeadm.x86_64 <span class="token number">1.21</span>.6-0 kuberneteskubeadm.x86_64 <span class="token number">1.21</span>.7-0 kubernetes<span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span>kubeadm.x86_64 <span class="token number">1.23</span>.3-0 kubernetes<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># yum list --showduplicates kubeadm --disableexcludes=kubernetes</span> 已加载插件：fastestmirror Repodata is over <span class="token number">2</span> weeks old. Install yum-cron? Or run: yum makecache fast Determining fastest mirrors 已安装的软件包 kubeadm.x86_64 <span class="token number">1.21</span>.0-0 @kubernetes 可安装的软件包 kubeadm.x86_64 <span class="token number">1.6</span>.0-0 kubernetes <span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span> kubeadm.x86_64 <span class="token number">1.21</span>.0-0 kubernetes kubeadm.x86_64 <span class="token number">1.21</span>.1-0 kubernetes kubeadm.x86_64 <span class="token number">1.21</span>.2-0 kubernetes kubeadm.x86_64 <span class="token number">1.21</span>.3-0 kubernetes kubeadm.x86_64 <span class="token number">1.21</span>.4-0 kubernetes kubeadm.x86_64 <span class="token number">1.21</span>.5-0 kubernetes kubeadm.x86_64 <span class="token number">1.21</span>.6-0 kubernetes kubeadm.x86_64 <span class="token number">1.21</span>.7-0 kubernetes <span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span> kubeadm.x86_64 <span class="token number">1.23</span>.3-0 kubernetes[root@k8scloude1 ~]# yum list --showduplicates kubeadm --disableexcludes=kubernetes 已加载插件：fastestmirror Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast Determining fastest mirrors 已安装的软件包 kubeadm.x86_64 1.21.0-0 @kubernetes 可安装的软件包 kubeadm.x86_64 1.6.0-0 kubernetes ...... kubeadm.x86_64 1.21.0-0 kubernetes kubeadm.x86_64 1.21.1-0 kubernetes kubeadm.x86_64 1.21.2-0 kubernetes kubeadm.x86_64 1.21.3-0 kubernetes kubeadm.x86_64 1.21.4-0 kubernetes kubeadm.x86_64 1.21.5-0 kubernetes kubeadm.x86_64 1.21.6-0 kubernetes kubeadm.x86_64 1.21.7-0 kubernetes ...... kubeadm.x86_64 1.23.3-0 kubernetes
升级kubeadm到1.21.9-0版本。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># yum install -y kubeadm-1.21.9-0 --disableexcludes=kubernetes</span>已加载插件：fastestmirrorbase <span class="token operator">|</span> <span class="token number">3.6</span> kB 00:00:00docker-ce-stable <span class="token operator">|</span> <span class="token number">3.5</span> kB 00:00:00epel <span class="token operator">|</span> <span class="token number">4.7</span> kB 00:00:00<span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span>更新完毕:kubeadm.x86_64 <span class="token number">0</span>:1.21.9-0完毕！<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># yum install -y kubeadm-1.21.9-0 --disableexcludes=kubernetes</span> 已加载插件：fastestmirror base <span class="token operator">|</span> <span class="token number">3.6</span> kB 00:00:00 docker-ce-stable <span class="token operator">|</span> <span class="token number">3.5</span> kB 00:00:00 epel <span class="token operator">|</span> <span class="token number">4.7</span> kB 00:00:00 <span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span> 更新完毕: kubeadm.x86_64 <span class="token number">0</span>:1.21.9-0 完毕！[root@k8scloude1 ~]# yum install -y kubeadm-1.21.9-0 --disableexcludes=kubernetes 已加载插件：fastestmirror base | 3.6 kB 00:00:00 docker-ce-stable | 3.5 kB 00:00:00 epel | 4.7 kB 00:00:00 ...... 更新完毕: kubeadm.x86_64 0:1.21.9-0 完毕！
kubeadm upgrade plan验证升级计划，COMPONENT CURRENT TARGET ：告诉我们组件可以从当前版本升级到的版本。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubeadm upgrade plan</span><span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> Making sure the configuration is correct:<span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> Reading configuration from the cluster<span class="token punctuation">..</span>.<span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> FYI: You can <span class="token function">look</span> at this config <span class="token function">file</span> with <span class="token string">'kubectl -n kube-system get cm kubeadm-config -o yaml'</span><span class="token punctuation">[</span>preflight<span class="token punctuation">]</span> Running pre-flight checks.<span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Running cluster health checks<span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Fetching available versions to upgrade to<span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> Cluster version: v1.21.0<span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> kubeadm version: v1.21.9I0402 <span class="token number">18</span>:09:32.079344 <span class="token number">16243</span> version.go:254<span class="token punctuation">]</span> remote version is much newer: v1.23.5<span class="token punctuation">;</span> falling back to: stable-1.21<span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> Target version: v1.21.11<span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> Latest version <span class="token keyword">in</span> the v1.21 series: v1.21.11Components that must be upgraded manually after you have upgraded the control plane with <span class="token string">'kubeadm upgrade apply'</span><span class="token builtin class-name">:</span>COMPONENT CURRENT TARGETkubelet <span class="token number">3</span> x v1.21.0 v1.21.11Upgrade to the latest version <span class="token keyword">in</span> the v1.21 series:COMPONENT CURRENT TARGETkube-apiserver v1.21.0 v1.21.11kube-controller-manager v1.21.0 v1.21.11kube-scheduler v1.21.0 v1.21.11kube-proxy v1.21.0 v1.21.11CoreDNS v1.8.0 v1.8.0etcd <span class="token number">3.4</span>.13-0 <span class="token number">3.4</span>.13-0You can now apply the upgrade by executing the following command:kubeadm upgrade apply v1.21.11Note: Before you can perform this upgrade, you have to update kubeadm to v1.21.11._____________________________________________________________________The table below shows the current state of component configs as understood by this version of kubeadm.Configs that have a <span class="token string">"yes"</span> mark <span class="token keyword">in</span> the <span class="token string">"MANUAL UPGRADE REQUIRED"</span> <span class="token function">column</span> require manual config upgrade orresetting to kubeadm defaults before a successful upgrade can be performed. The version to manuallyupgrade to is denoted <span class="token keyword">in</span> the <span class="token string">"PREFERRED VERSION"</span> column.API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIREDkubeproxy.config.k8s.io v1alpha1 v1alpha1 nokubelet.config.k8s.io v1beta1 v1beta1 no_____________________________________________________________________<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubeadm upgrade plan</span> <span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> Making sure the configuration is correct: <span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> Reading configuration from the cluster<span class="token punctuation">..</span>. <span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> FYI: You can <span class="token function">look</span> at this config <span class="token function">file</span> with <span class="token string">'kubectl -n kube-system get cm kubeadm-config -o yaml'</span> <span class="token punctuation">[</span>preflight<span class="token punctuation">]</span> Running pre-flight checks. <span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Running cluster health checks <span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Fetching available versions to upgrade to <span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> Cluster version: v1.21.0 <span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> kubeadm version: v1.21.9 I0402 <span class="token number">18</span>:09:32.079344 <span class="token number">16243</span> version.go:254<span class="token punctuation">]</span> remote version is much newer: v1.23.5<span class="token punctuation">;</span> falling back to: stable-1.21 <span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> Target version: v1.21.11 <span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> Latest version <span class="token keyword">in</span> the v1.21 series: v1.21.11 Components that must be upgraded manually after you have upgraded the control plane with <span class="token string">'kubeadm upgrade apply'</span><span class="token builtin class-name">:</span> COMPONENT CURRENT TARGET kubelet <span class="token number">3</span> x v1.21.0 v1.21.11 Upgrade to the latest version <span class="token keyword">in</span> the v1.21 series: COMPONENT CURRENT TARGET kube-apiserver v1.21.0 v1.21.11 kube-controller-manager v1.21.0 v1.21.11 kube-scheduler v1.21.0 v1.21.11 kube-proxy v1.21.0 v1.21.11 CoreDNS v1.8.0 v1.8.0 etcd <span class="token number">3.4</span>.13-0 <span class="token number">3.4</span>.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.21.11 Note: Before you can perform this upgrade, you have to update kubeadm to v1.21.11. _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a <span class="token string">"yes"</span> mark <span class="token keyword">in</span> the <span class="token string">"MANUAL UPGRADE REQUIRED"</span> <span class="token function">column</span> require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted <span class="token keyword">in</span> the <span class="token string">"PREFERRED VERSION"</span> column. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no _____________________________________________________________________[root@k8scloude1 ~]# kubeadm upgrade plan [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade] Fetching available versions to upgrade to [upgrade/versions] Cluster version: v1.21.0 [upgrade/versions] kubeadm version: v1.21.9 I0402 18:09:32.079344 16243 version.go:254] remote version is much newer: v1.23.5; falling back to: stable-1.21 [upgrade/versions] Target version: v1.21.11 [upgrade/versions] Latest version in the v1.21 series: v1.21.11 Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT CURRENT TARGET kubelet 3 x v1.21.0 v1.21.11 Upgrade to the latest version in the v1.21 series: COMPONENT CURRENT TARGET kube-apiserver v1.21.0 v1.21.11 kube-controller-manager v1.21.0 v1.21.11 kube-scheduler v1.21.0 v1.21.11 kube-proxy v1.21.0 v1.21.11 CoreDNS v1.8.0 v1.8.0 etcd 3.4.13-0 3.4.13-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.21.11 Note: Before you can perform this upgrade, you have to update kubeadm to v1.21.11. _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denoted in the "PREFERRED VERSION" column. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no _____________________________________________________________________
4.2 升级各个组件
上一步是升级kubeadm ,接下来升级各个组件（kube-apiserver，kube-controller-manager等等），
kubeadm upgrade apply v1.21.9升级各个组件到1.21.9版本，如果etcd这个组件不想升级，可以加上选项:kubeadm upgrade apply v1.21.9 –etcd-upgrade=false。
可以提前drain节点，后面drain也可以。
腾空节点：通过将节点标记为不可调度并腾空节点为节点作升级准备：kubectl drain
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl drain k8scloude1 --ignore-daemonsets</span>node/k8scloude1 cordonederror: unable to drain <span class="token function">node</span> <span class="token string">"k8scloude1"</span>, aborting command<span class="token punctuation">..</span>.There are pending nodes to be drained:k8scloude1error: cannot delete Pods with <span class="token builtin class-name">local</span> storage <span class="token punctuation">(</span>use --delete-emptydir-data to override<span class="token punctuation">)</span>: kube-system/metrics-server-bcfb98c76-j4gs8, kubernetes-dashboard/dashboard-metrics-scraper-7f458d9467-9knf9<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl drain k8scloude1 --ignore-daemonsets</span> node/k8scloude1 cordoned error: unable to drain <span class="token function">node</span> <span class="token string">"k8scloude1"</span>, aborting command<span class="token punctuation">..</span>. There are pending nodes to be drained: k8scloude1 error: cannot delete Pods with <span class="token builtin class-name">local</span> storage <span class="token punctuation">(</span>use --delete-emptydir-data to override<span class="token punctuation">)</span>: kube-system/metrics-server-bcfb98c76-j4gs8, kubernetes-dashboard/dashboard-metrics-scraper-7f458d9467-9knf9[root@k8scloude1 ~]# kubectl drain k8scloude1 --ignore-daemonsets node/k8scloude1 cordoned error: unable to drain node "k8scloude1", aborting command... There are pending nodes to be drained: k8scloude1 error: cannot delete Pods with local storage (use --delete-emptydir-data to override): kube-system/metrics-server-bcfb98c76-j4gs8, kubernetes-dashboard/dashboard-metrics-scraper-7f458d9467-9knf9
因为有本地数据，需要加–delete-emptydir-data选项。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl drain k8scloude1 --ignore-daemonsets --delete-emptydir-data </span>node/k8scloude1 already cordonedWARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-bdlgm, kube-system/kube-proxy-zxlk9, metallb-system/speaker-jfccbevicting pod kubernetes-dashboard/dashboard-metrics-scraper-7f458d9467-9knf9evicting pod kube-system/coredns-545d6fc579-7wm95evicting pod kube-system/coredns-545d6fc579-87q8jevicting pod kube-system/metrics-server-bcfb98c76-j4gs8pod/dashboard-metrics-scraper-7f458d9467-9knf9 evictedpod/metrics-server-bcfb98c76-j4gs8 evictedpod/coredns-545d6fc579-7wm95 evictedpod/coredns-545d6fc579-87q8j evictednode/k8scloude1 evicted<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl drain k8scloude1 --ignore-daemonsets --delete-emptydir-data </span> node/k8scloude1 already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-bdlgm, kube-system/kube-proxy-zxlk9, metallb-system/speaker-jfccb evicting pod kubernetes-dashboard/dashboard-metrics-scraper-7f458d9467-9knf9 evicting pod kube-system/coredns-545d6fc579-7wm95 evicting pod kube-system/coredns-545d6fc579-87q8j evicting pod kube-system/metrics-server-bcfb98c76-j4gs8 pod/dashboard-metrics-scraper-7f458d9467-9knf9 evicted pod/metrics-server-bcfb98c76-j4gs8 evicted pod/coredns-545d6fc579-7wm95 evicted pod/coredns-545d6fc579-87q8j evicted node/k8scloude1 evicted[root@k8scloude1 ~]# kubectl drain k8scloude1 --ignore-daemonsets --delete-emptydir-data node/k8scloude1 already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-bdlgm, kube-system/kube-proxy-zxlk9, metallb-system/speaker-jfccb evicting pod kubernetes-dashboard/dashboard-metrics-scraper-7f458d9467-9knf9 evicting pod kube-system/coredns-545d6fc579-7wm95 evicting pod kube-system/coredns-545d6fc579-87q8j evicting pod kube-system/metrics-server-bcfb98c76-j4gs8 pod/dashboard-metrics-scraper-7f458d9467-9knf9 evicted pod/metrics-server-bcfb98c76-j4gs8 evicted pod/coredns-545d6fc579-7wm95 evicted pod/coredns-545d6fc579-87q8j evicted node/k8scloude1 evicted
升级各个组件，–etcd-upgrade=false表示etcd数据库不升级。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubeadm upgrade apply v1.21.9 --etcd-upgrade=false</span><span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> Making sure the configuration is correct:<span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> Reading configuration from the cluster<span class="token punctuation">..</span>.<span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> FYI: You can <span class="token function">look</span> at this config <span class="token function">file</span> with <span class="token string">'kubectl -n kube-system get cm kubeadm-config -o yaml'</span><span class="token punctuation">[</span>preflight<span class="token punctuation">]</span> Running pre-flight checks.<span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Running cluster health checks<span class="token punctuation">[</span>upgrade/version<span class="token punctuation">]</span> You have chosen to change the cluster version to <span class="token string">"v1.21.9"</span><span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> Cluster version: v1.21.0<span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> kubeadm version: v1.21.9<span class="token punctuation">[</span>upgrade/confirm<span class="token punctuation">]</span> Are you sure you want to proceed with the upgrade? <span class="token punctuation">[</span>y/N<span class="token punctuation">]</span>: y<span class="token punctuation">[</span>upgrade/prepull<span class="token punctuation">]</span> Pulling images required <span class="token keyword">for</span> setting up a Kubernetes cluster<span class="token punctuation">[</span>upgrade/prepull<span class="token punctuation">]</span> This might take a minute or two, depending on the speed of your internet connection<span class="token punctuation">[</span>upgrade/prepull<span class="token punctuation">]</span> You can also perform this action <span class="token keyword">in</span> beforehand using <span class="token string">'kubeadm config images pull'</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">[</span>bootstrap-token<span class="token punctuation">]</span> configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token<span class="token punctuation">[</span>bootstrap-token<span class="token punctuation">]</span> configured RBAC rules to allow certificate rotation <span class="token keyword">for</span> all <span class="token function">node</span> client certificates <span class="token keyword">in</span> the cluster<span class="token punctuation">[</span>addons<span class="token punctuation">]</span> Applied essential addon: CoreDNS<span class="token punctuation">[</span>addons<span class="token punctuation">]</span> Applied essential addon: kube-proxy<span class="token punctuation">[</span>upgrade/successful<span class="token punctuation">]</span> SUCCESS<span class="token operator">!</span> Your cluster was upgraded to <span class="token string">"v1.21.9"</span><span class="token builtin class-name">.</span> Enjoy<span class="token operator">!</span><span class="token punctuation">[</span>upgrade/kubelet<span class="token punctuation">]</span> Now that your control plane is upgraded, please proceed with upgrading your kubelets <span class="token keyword">if</span> you haven't already <span class="token keyword">done</span> so.<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubeadm upgrade apply v1.21.9 --etcd-upgrade=false</span> <span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> Making sure the configuration is correct: <span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> Reading configuration from the cluster<span class="token punctuation">..</span>. <span class="token punctuation">[</span>upgrade/config<span class="token punctuation">]</span> FYI: You can <span class="token function">look</span> at this config <span class="token function">file</span> with <span class="token string">'kubectl -n kube-system get cm kubeadm-config -o yaml'</span> <span class="token punctuation">[</span>preflight<span class="token punctuation">]</span> Running pre-flight checks. <span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Running cluster health checks <span class="token punctuation">[</span>upgrade/version<span class="token punctuation">]</span> You have chosen to change the cluster version to <span class="token string">"v1.21.9"</span> <span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> Cluster version: v1.21.0 <span class="token punctuation">[</span>upgrade/versions<span class="token punctuation">]</span> kubeadm version: v1.21.9 <span class="token punctuation">[</span>upgrade/confirm<span class="token punctuation">]</span> Are you sure you want to proceed with the upgrade? <span class="token punctuation">[</span>y/N<span class="token punctuation">]</span>: y <span class="token punctuation">[</span>upgrade/prepull<span class="token punctuation">]</span> Pulling images required <span class="token keyword">for</span> setting up a Kubernetes cluster <span class="token punctuation">[</span>upgrade/prepull<span class="token punctuation">]</span> This might take a minute or two, depending on the speed of your internet connection <span class="token punctuation">[</span>upgrade/prepull<span class="token punctuation">]</span> You can also perform this action <span class="token keyword">in</span> beforehand using <span class="token string">'kubeadm config images pull'</span> <span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span> <span class="token punctuation">[</span>bootstrap-token<span class="token punctuation">]</span> configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token <span class="token punctuation">[</span>bootstrap-token<span class="token punctuation">]</span> configured RBAC rules to allow certificate rotation <span class="token keyword">for</span> all <span class="token function">node</span> client certificates <span class="token keyword">in</span> the cluster <span class="token punctuation">[</span>addons<span class="token punctuation">]</span> Applied essential addon: CoreDNS <span class="token punctuation">[</span>addons<span class="token punctuation">]</span> Applied essential addon: kube-proxy <span class="token punctuation">[</span>upgrade/successful<span class="token punctuation">]</span> SUCCESS<span class="token operator">!</span> Your cluster was upgraded to <span class="token string">"v1.21.9"</span><span class="token builtin class-name">.</span> Enjoy<span class="token operator">!</span> <span class="token punctuation">[</span>upgrade/kubelet<span class="token punctuation">]</span> Now that your control plane is upgraded, please proceed with upgrading your kubelets <span class="token keyword">if</span> you haven't already <span class="token keyword">done</span> so.[root@k8scloude1 ~]# kubeadm upgrade apply v1.21.9 --etcd-upgrade=false [upgrade/config] Making sure the configuration is correct: [upgrade/config] Reading configuration from the cluster... [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks. [upgrade] Running cluster health checks [upgrade/version] You have chosen to change the cluster version to "v1.21.9" [upgrade/versions] Cluster version: v1.21.0 [upgrade/versions] kubeadm version: v1.21.9 [upgrade/confirm] Are you sure you want to proceed with the upgrade? [y/N]: y [upgrade/prepull] Pulling images required for setting up a Kubernetes cluster [upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection [upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull' ...... [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.21.9". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
现在k8scloude1节点是不可调度状态的。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span>NAME STATUS ROLES AGE VERSIONk8scloude1 Ready,SchedulingDisabled control-plane,master 83d v1.21.0k8scloude2 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span> NAME STATUS ROLES AGE VERSION k8scloude1 Ready,SchedulingDisabled control-plane,master 83d v1.21.0 k8scloude2 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0 k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0[root@k8scloude1 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8scloude1 Ready,SchedulingDisabled control-plane,master 83d v1.21.0 k8scloude2 Ready <none> 83d v1.21.0 k8scloude3 Ready <none> 83d v1.21.0
解除k8scloude1节点的保护：通过将节点标记为可调度，让其重新上线。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl uncordon k8scloude1</span>node/k8scloude1 uncordoned<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl uncordon k8scloude1</span> node/k8scloude1 uncordoned[root@k8scloude1 ~]# kubectl uncordon k8scloude1 node/k8scloude1 uncordoned
现在k8scloude1节点是Ready状态的。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span>NAME STATUS ROLES AGE VERSIONk8scloude1 Ready control-plane,master 83d v1.21.0k8scloude2 NotReady <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span> NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.0 k8scloude2 NotReady <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0 k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0[root@k8scloude1 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.0 k8scloude2 NotReady <none> 83d v1.21.0 k8scloude3 Ready <none> 83d v1.21.0
4.3 升级 kubelet 和 kubectl
升级 kubelet 和 kubectl到1.21.9版本：yum install -y kubelet-1.23.x-0 kubectl-1.23.x-0 –disableexcludes=kubernetes。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># yum install -y kubelet-1.21.9-0 kubectl-1.21.9-0 --disableexcludes=kubernetes</span>已加载插件：fastestmirrorLoading mirror speeds from cached hostfile正在解决依赖关系--<span class="token operator">></span> 正在检查事务---<span class="token operator">></span> 软件包 kubectl.x86_64.0.1.21.0-0 将被 升级<span class="token punctuation">..</span><span class="token punctuation">..</span>.更新完毕:kubectl.x86_64 <span class="token number">0</span>:1.21.9-0 kubelet.x86_64 <span class="token number">0</span>:1.21.9-0完毕！<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># yum install -y kubelet-1.21.9-0 kubectl-1.21.9-0 --disableexcludes=kubernetes</span> 已加载插件：fastestmirror Loading mirror speeds from cached hostfile 正在解决依赖关系 --<span class="token operator">></span> 正在检查事务 ---<span class="token operator">></span> 软件包 kubectl.x86_64.0.1.21.0-0 将被 升级 <span class="token punctuation">..</span><span class="token punctuation">..</span>. 更新完毕: kubectl.x86_64 <span class="token number">0</span>:1.21.9-0 kubelet.x86_64 <span class="token number">0</span>:1.21.9-0 完毕！[root@k8scloude1 ~]# yum install -y kubelet-1.21.9-0 kubectl-1.21.9-0 --disableexcludes=kubernetes 已加载插件：fastestmirror Loading mirror speeds from cached hostfile 正在解决依赖关系 --> 正在检查事务 ---> 软件包 kubectl.x86_64.0.1.21.0-0 将被 升级 ..... 更新完毕: kubectl.x86_64 0:1.21.9-0 kubelet.x86_64 0:1.21.9-0 完毕！
重新加载配置文件并重启 kubelet。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># systemctl daemon-reload ;systemctl restart kubelet</span><span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># systemctl daemon-reload ;systemctl restart kubelet</span>[root@k8scloude1 ~]# systemctl daemon-reload ;systemctl restart kubelet
此时k8scloude1节点的版本就变为v1.21.9了，k8s集群的master节点升级成功，如果有多个master，则步骤一样，但是第二个master节点不需要执行kubeadm upgrade apply v1.21.9命令，第二台master节点把kubeadm upgrade apply v1.21.9变为kubeadm upgrade node。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span>NAME STATUS ROLES AGE VERSIONk8scloude1 Ready control-plane,master 83d v1.21.9k8scloude2 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span> NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0 k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0[root@k8scloude1 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 Ready <none> 83d v1.21.0 k8scloude3 Ready <none> 83d v1.21.0
五.升级worker工作节点
5.1 升级kubeadm
升级kubeadm到1.21.9版本。
<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># yum install -y kubeadm-1.21.9-0 --disableexcludes=kubernetes</span>已加载插件：fastestmirrorbase <span class="token operator">|</span> <span class="token number">3.6</span> kB 00:00:00docker-ce-stable <span class="token operator">|</span> <span class="token number">3.5</span> kB 00:00:00<span class="token punctuation">..</span><span class="token punctuation">..</span>.更新完毕:kubeadm.x86_64 <span class="token number">0</span>:1.21.9-0完毕！<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># yum install -y kubeadm-1.21.9-0 --disableexcludes=kubernetes</span> 已加载插件：fastestmirror base <span class="token operator">|</span> <span class="token number">3.6</span> kB 00:00:00 docker-ce-stable <span class="token operator">|</span> <span class="token number">3.5</span> kB 00:00:00 <span class="token punctuation">..</span><span class="token punctuation">..</span>. 更新完毕: kubeadm.x86_64 <span class="token number">0</span>:1.21.9-0 完毕！[root@k8scloude2 ~]# yum install -y kubeadm-1.21.9-0 --disableexcludes=kubernetes 已加载插件：fastestmirror base | 3.6 kB 00:00:00 docker-ce-stable | 3.5 kB 00:00:00 ..... 更新完毕: kubeadm.x86_64 0:1.21.9-0 完毕！
通过将节点标记为不可调度并腾空节点，为节点作升级准备。
<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl drain k8scloude2 --ignore-daemonsets</span>node/k8scloude2 cordonederror: unable to drain <span class="token function">node</span> <span class="token string">"k8scloude2"</span>, aborting command<span class="token punctuation">..</span>.There are pending nodes to be drained:k8scloude2error: cannot delete Pods with <span class="token builtin class-name">local</span> storage <span class="token punctuation">(</span>use --delete-emptydir-data to override<span class="token punctuation">)</span>: kubernetes-dashboard/kubernetes-dashboard-7bf6f979bf-c4ckm<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl drain k8scloude2 --ignore-daemonsets</span> node/k8scloude2 cordoned error: unable to drain <span class="token function">node</span> <span class="token string">"k8scloude2"</span>, aborting command<span class="token punctuation">..</span>. There are pending nodes to be drained: k8scloude2 error: cannot delete Pods with <span class="token builtin class-name">local</span> storage <span class="token punctuation">(</span>use --delete-emptydir-data to override<span class="token punctuation">)</span>: kubernetes-dashboard/kubernetes-dashboard-7bf6f979bf-c4ckm[root@k8scloude2 ~]# kubectl drain k8scloude2 --ignore-daemonsets node/k8scloude2 cordoned error: unable to drain node "k8scloude2", aborting command... There are pending nodes to be drained: k8scloude2 error: cannot delete Pods with local storage (use --delete-emptydir-data to override): kubernetes-dashboard/kubernetes-dashboard-7bf6f979bf-c4ckm
因为有本地数据，需要加–delete-emptydir-data选项。
<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl drain k8scloude2 --ignore-daemonsets --delete-emptydir-data</span>node/k8scloude2 already cordonedWARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-nsbfs, kube-system/kube-proxy-r289t, metallb-system/speaker-nkrgkevicting pod volume/nfs-client-provisioner-76c576954d-5x7t2evicting pod kube-system/calico-kube-controllers-6b9fbfff44-dh5vnevicting pod ingress-nginx/ingress-nginx-admission-patch-hd7p4evicting pod kube-system/kuboard-78dccb7d9f-pgbdx<span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span>I0402 <span class="token number">18</span>:47:15.461157 <span class="token number">15424</span> request.go:668<span class="token punctuation">]</span> Waited <span class="token keyword">for</span> <span class="token number">1</span>.039246436s due to client-side throttling, not priority and fairness, request: GET:https://192.168.110.130:6443/api/v1/namespaces/kube-system/pods/coredns-59d64cd4d4-7jkphpod/ingress-nginx-controller-59b8bf5fdc-n89sm evictedI0402 <span class="token number">18</span>:47:26.461436 <span class="token number">15424</span> request.go:668<span class="token punctuation">]</span> Waited <span class="token keyword">for</span> <span class="token number">1</span>.036011034s due to client-side throttling, not priority and fairness, request: GET:https://192.168.110.130:6443/api/v1/namespaces/volume/pods/nfs-client-provisioner-76c576954d-5x7t2pod/calico-kube-controllers-6b9fbfff44-dh5vn evictedpod/kubernetes-dashboard-7bf6f979bf-c4ckm evictedpod/kuboard-78dccb7d9f-pgbdx evictedpod/coredns-59d64cd4d4-7jkph evicted<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl drain k8scloude2 --ignore-daemonsets --delete-emptydir-data</span> node/k8scloude2 already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-nsbfs, kube-system/kube-proxy-r289t, metallb-system/speaker-nkrgk evicting pod volume/nfs-client-provisioner-76c576954d-5x7t2 evicting pod kube-system/calico-kube-controllers-6b9fbfff44-dh5vn evicting pod ingress-nginx/ingress-nginx-admission-patch-hd7p4 evicting pod kube-system/kuboard-78dccb7d9f-pgbdx <span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span> I0402 <span class="token number">18</span>:47:15.461157 <span class="token number">15424</span> request.go:668<span class="token punctuation">]</span> Waited <span class="token keyword">for</span> <span class="token number">1</span>.039246436s due to client-side throttling, not priority and fairness, request: GET:https://192.168.110.130:6443/api/v1/namespaces/kube-system/pods/coredns-59d64cd4d4-7jkph pod/ingress-nginx-controller-59b8bf5fdc-n89sm evicted I0402 <span class="token number">18</span>:47:26.461436 <span class="token number">15424</span> request.go:668<span class="token punctuation">]</span> Waited <span class="token keyword">for</span> <span class="token number">1</span>.036011034s due to client-side throttling, not priority and fairness, request: GET:https://192.168.110.130:6443/api/v1/namespaces/volume/pods/nfs-client-provisioner-76c576954d-5x7t2 pod/calico-kube-controllers-6b9fbfff44-dh5vn evicted pod/kubernetes-dashboard-7bf6f979bf-c4ckm evicted pod/kuboard-78dccb7d9f-pgbdx evicted pod/coredns-59d64cd4d4-7jkph evicted[root@k8scloude2 ~]# kubectl drain k8scloude2 --ignore-daemonsets --delete-emptydir-data node/k8scloude2 already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-nsbfs, kube-system/kube-proxy-r289t, metallb-system/speaker-nkrgk evicting pod volume/nfs-client-provisioner-76c576954d-5x7t2 evicting pod kube-system/calico-kube-controllers-6b9fbfff44-dh5vn evicting pod ingress-nginx/ingress-nginx-admission-patch-hd7p4 evicting pod kube-system/kuboard-78dccb7d9f-pgbdx ...... I0402 18:47:15.461157 15424 request.go:668] Waited for 1.039246436s due to client-side throttling, not priority and fairness, request: GET:https://192.168.110.130:6443/api/v1/namespaces/kube-system/pods/coredns-59d64cd4d4-7jkph pod/ingress-nginx-controller-59b8bf5fdc-n89sm evicted I0402 18:47:26.461436 15424 request.go:668] Waited for 1.036011034s due to client-side throttling, not priority and fairness, request: GET:https://192.168.110.130:6443/api/v1/namespaces/volume/pods/nfs-client-provisioner-76c576954d-5x7t2 pod/calico-kube-controllers-6b9fbfff44-dh5vn evicted pod/kubernetes-dashboard-7bf6f979bf-c4ckm evicted pod/kuboard-78dccb7d9f-pgbdx evicted pod/coredns-59d64cd4d4-7jkph evicted
k8scloude2节点现在是不可调度状态。
<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span>NAME STATUS ROLES AGE VERSIONk8scloude1 Ready control-plane,master 83d v1.21.9k8scloude2 NotReady,SchedulingDisabled <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span> NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 NotReady,SchedulingDisabled <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0 k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0[root@k8scloude2 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 NotReady,SchedulingDisabled <none> 83d v1.21.0 k8scloude3 Ready <none> 83d v1.21.0
对于工作节点， kubeadm upgrade node 命令会升级本地的 kubelet 配置。
<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubeadm upgrade node</span><span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Reading configuration from the cluster<span class="token punctuation">..</span>.<span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> FYI: You can <span class="token function">look</span> at this config <span class="token function">file</span> with <span class="token string">'kubectl -n kube-system get cm kubeadm-config -o yaml'</span><span class="token punctuation">[</span>preflight<span class="token punctuation">]</span> Running pre-flight checks<span class="token punctuation">[</span>preflight<span class="token punctuation">]</span> Skipping prepull. Not a control plane node.<span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Skipping phase. Not a control plane node.<span class="token punctuation">[</span>kubelet-start<span class="token punctuation">]</span> Writing kubelet configuration to <span class="token function">file</span> <span class="token string">"/var/lib/kubelet/config.yaml"</span><span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> The configuration <span class="token keyword">for</span> this <span class="token function">node</span> was successfully updated<span class="token operator">!</span><span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Now you should go ahead and upgrade the kubelet package using your package manager.<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubeadm upgrade node</span> <span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Reading configuration from the cluster<span class="token punctuation">..</span>. <span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> FYI: You can <span class="token function">look</span> at this config <span class="token function">file</span> with <span class="token string">'kubectl -n kube-system get cm kubeadm-config -o yaml'</span> <span class="token punctuation">[</span>preflight<span class="token punctuation">]</span> Running pre-flight checks <span class="token punctuation">[</span>preflight<span class="token punctuation">]</span> Skipping prepull. Not a control plane node. <span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Skipping phase. Not a control plane node. <span class="token punctuation">[</span>kubelet-start<span class="token punctuation">]</span> Writing kubelet configuration to <span class="token function">file</span> <span class="token string">"/var/lib/kubelet/config.yaml"</span> <span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> The configuration <span class="token keyword">for</span> this <span class="token function">node</span> was successfully updated<span class="token operator">!</span> <span class="token punctuation">[</span>upgrade<span class="token punctuation">]</span> Now you should go ahead and upgrade the kubelet package using your package manager.[root@k8scloude2 ~]# kubeadm upgrade node [upgrade] Reading configuration from the cluster... [upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [preflight] Running pre-flight checks [preflight] Skipping prepull. Not a control plane node. [upgrade] Skipping phase. Not a control plane node. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [upgrade] The configuration for this node was successfully updated! [upgrade] Now you should go ahead and upgrade the kubelet package using your package manager.
通过将k8scloude2节点标记为可调度，让其重新上线。
<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl uncordon k8scloude2</span>node/k8scloude2 uncordoned<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span>NAME STATUS ROLES AGE VERSIONk8scloude1 Ready control-plane,master 83d v1.21.9k8scloude2 NotReady <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl uncordon k8scloude2</span> node/k8scloude2 uncordoned <span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get node</span> NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 NotReady <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0 k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0[root@k8scloude2 ~]# kubectl uncordon k8scloude2 node/k8scloude2 uncordoned [root@k8scloude2 ~]# kubectl get node NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 NotReady <none> 83d v1.21.0 k8scloude3 Ready <none> 83d v1.21.0
5.2 升级kubelet和kubectl
升级kubelet和kubectl到1.21.9版本。
<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># yum install -y kubelet-1.21.9-0 kubectl-1.21.9-0 --disableexcludes=kubernetes</span>已加载插件：fastestmirrorLoading mirror speeds from cached hostfile正在解决依赖关系--<span class="token operator">></span> 正在检查事务---<span class="token operator">></span> 软件包 kubectl.x86_64.0.1.21.0-0 将被 升级<span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span>验证中 <span class="token builtin class-name">:</span> kubectl-1.21.0-0.x86_64 <span class="token number">4</span>/4更新完毕:kubectl.x86_64 <span class="token number">0</span>:1.21.9-0 kubelet.x86_64 <span class="token number">0</span>:1.21.9-0完毕！<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># yum install -y kubelet-1.21.9-0 kubectl-1.21.9-0 --disableexcludes=kubernetes</span> 已加载插件：fastestmirror Loading mirror speeds from cached hostfile 正在解决依赖关系 --<span class="token operator">></span> 正在检查事务 ---<span class="token operator">></span> 软件包 kubectl.x86_64.0.1.21.0-0 将被 升级 <span class="token punctuation">..</span><span class="token punctuation">..</span><span class="token punctuation">..</span> 验证中 <span class="token builtin class-name">:</span> kubectl-1.21.0-0.x86_64 <span class="token number">4</span>/4 更新完毕: kubectl.x86_64 <span class="token number">0</span>:1.21.9-0 kubelet.x86_64 <span class="token number">0</span>:1.21.9-0 完毕！[root@k8scloude2 ~]# yum install -y kubelet-1.21.9-0 kubectl-1.21.9-0 --disableexcludes=kubernetes 已加载插件：fastestmirror Loading mirror speeds from cached hostfile 正在解决依赖关系 --> 正在检查事务 ---> 软件包 kubectl.x86_64.0.1.21.0-0 将被 升级 ...... 验证中 : kubectl-1.21.0-0.x86_64 4/4 更新完毕: kubectl.x86_64 0:1.21.9-0 kubelet.x86_64 0:1.21.9-0 完毕！
重新加载配置文件并重启kubelet。
<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># systemctl daemon-reload ;systemctl restart kubelet</span><span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get nodes</span>NAME STATUS ROLES AGE VERSIONk8scloude1 Ready control-plane,master 83d v1.21.9k8scloude2 NotReady <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0<span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># systemctl daemon-reload ;systemctl restart kubelet</span> <span class="token punctuation">[</span>root@k8scloude2 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get nodes</span> NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 NotReady <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0 k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.0[root@k8scloude2 ~]# systemctl daemon-reload ;systemctl restart kubelet [root@k8scloude2 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 NotReady <none> 83d v1.21.0 k8scloude3 Ready <none> 83d v1.21.0
k8scloude3这个worker工作节点的升级步骤和k8scloude2节点一模一样。
k8scloude3节点升级之后，整个Kubernetes(k8s) 集群就升级完毕了，版本都是v1.21.9。
<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get nodes</span>NAME STATUS ROLES AGE VERSIONk8scloude1 Ready control-plane,master 83d v1.21.9k8scloude2 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.9k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.9<span class="token punctuation">[</span>root@k8scloude1 ~<span class="token punctuation">]</span><span class="token comment"># kubectl get nodes</span> NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.9 k8scloude3 Ready <span class="token operator"><</span>none<span class="token operator">></span> 83d v1.21.9[root@k8scloude1 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8scloude1 Ready control-plane,master 83d v1.21.9 k8scloude2 Ready <none> 83d v1.21.9 k8scloude3 Ready <none> 83d v1.21.9
六.总结
升级Kubernetes(k8s) 集群是确保集群稳定性和安全性的重要步骤。请记住，在执行任何升级操作之前，务必备份重要数据和配置文件。
本博客涵盖了在 CentOS 7.4 上升级 Kubernetes(k8s) 集群的主要步骤，但具体操作过程可能因环境和配置的不同而有所差异。
© 版权声明
文章版权归作者所有，未经允许请勿转载，侵权请联系 admin@trc20.tw 删除。
THE END
