关于我
Centos搭建Docker和Kubernetes(一)
搭建Docker+Kubernetes环境，然后部署微服务，最后开放对外访问。
由于网络问题，可能下载安装过程中会失败，耐心多安装几次就OK了。
安装一定要有耐心！！！安装一定要有耐心！！！安装一定要有耐心！！！
环境
Centos：7.5
Docker：19.03
Kubernetes：1.1.18
准备两台CentOS机器，这里为了测试，只准备两台，1台做Master，1台做Worker。
安装Docker
使用脚本安装Docker
使用root登录Centos
确保yum包更新到最新
yum update
执行Docker安装脚本
curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
执行这个脚本会添加 docker.repo 源并安装 Docker。
启动Docker并且设置为开机启动
systemctl start docker
systemctl enable docker
至此Docker在Centos系统的安装已完成。
配置镜像加速器
鉴于国内网络问题，拉取Docker镜像十分缓慢，我们需要配置加速器来提速。
增加配置：
vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://847pb1vj.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
生效配置、重启Docker：
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
关闭防火墙
systemctl stop firewalld # 关闭服务
systemctl disable firewalld # 禁用服务
禁用SELinux
修改/etc/selinux/config，设置SELINUX=disabled，重启机器。
禁用交换分区
swapoff -a
vim /etc/sysconfig/kubelet
修改为：KUBELET_EXTRA_ARGS="--fail-swap-on=false"
安装Kubernetes
安装kubelet、kubeadm、kubectl、dashboard、添加woker。
添加源
由于国内网络原因，官方文档中的地址不可用，本文替换为阿里云镜像地址：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
安装
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable kubelet && systemctl start kubelet
修改网络配置
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
注意： 至此，以上的全部操作， 在Worker机器上也需要执行，包括安装Docker。
初始化Master
生成初始化文件
kubeadm config print init-defaults > kubeadm-init.yaml
该文件有3处需要修改：
- 将
advertiseAddress: 1.2.3.4修改为本机地址（ 注意是内网地址 ） - 将
imageRepository: k8s.gcr.io修改为imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers - nodeRegistration:name，改为k8s-master（woker节点的话，则修改为别的名称）
修改完毕之后：
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.0.237
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8s-master
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.15.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
scheduler: {}
下载镜像
kubeadm config images pull --config kubeadm-init.yaml
输入如下信息：
执行初始化
kubeadm init --config kubeadm-init.yaml
等待执行完毕后, 会输出如下内容：
W0328 21:31:14.954124 17338 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.0
[preflight] Running pre-flight checks
[WARNING Hostname]: hostname "k8s-master" could not be reached
[WARNING Hostname]: hostname "k8s-master": lookup k8s-master on 100.125.1.250:53: no such host
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.237]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.237 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master localhost] and IPs [192.168.0.237 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0328 21:31:18.364428 17338 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0328 21:31:18.365138 17338 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.501912 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.237:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:964c7b0c007ce17c979e631da17ad047dfa3bad76e407b6ee76d729ecf3cd9c7
最后两行需要保存下来， kubeadm join ...是Worker节点加入所需要执行的命令。
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
注意此处：可能安装时一直等待，最后超时，错误如下。
可能是因为kubeadm-init.yaml中advertiseAddress没配置成内网，导致网络问题。
接下来配置环境，让当前用户可以执行kubectl命令:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
测试一下：此处的NotReady是因为网络还没配置。
[root@ecs-babc-0004 kubernetes]$ kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master NotReady master 2m25s v1.17.3
配置网络
安装calico网络插件：
[root@ecs-babc-0004 ~]$ wget https://docs.projectcalico.org/v3.9/manifests/calico.yaml
[root@ecs-babc-0004 ~]$ cat kubeadm-init.yaml | grep serviceSubnet:
serviceSubnet: 10.96.0.0/12
vim calico.yaml，将192.168.0.0/16修改为10.96.0.0/12
需要注意的是, calico.yaml中的IP和kubeadm-init.yaml需要保持一致, 要么初始化前修改kubeadm-init.yaml, 要么初始化后修改calico.yaml.
执行kubectl apply -f calico.yaml初始化网络。
[root@ecs-babc-0004 kubernetes]# kubectl apply -f calico.yaml
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
过几分钟再查看node信息，master的状态已经是Ready了。
[root@ecs-babc-0004 kubernetes]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 11m v1.18.0
此处有坑，半支烟被坑了好久，后来才明白，是因为拉取不到calico/cni:v3.8.8镜像，或者拉取很慢，找到坑之后，果断换版本
calico/cni:v3.9.5******，分分钟就OK了。
以下是排查过程：
首先执行kubectl get pods -n kube-system发现pods一直pending，发现镜像一直在pulling，有时1个小时都拉取不下来。
异常情况：
而正常情况应该是：
[root@ecs-babc-0004 kubernetes]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-5fc5dbfc47-4gvcb 0/1 Pending 0 43s
calico-node-gvnnc 0/1 Init:2/3 0 43s
coredns-546565776c-sx5rj 0/1 Pending 0 6m52s
coredns-546565776c-xtb92 0/1 Pending 0 6m52s
etcd-k8s-master 1/1 Running 0 6m51s
kube-apiserver-k8s-master 1/1 Running 0 6m51s
kube-controller-manager-k8s-master 1/1 Running 0 6m51s
kube-proxy-7jk4h 1/1 Running 0 6m52s
kube-scheduler-k8s-master 1/1 Running 0 6m50s
再执行kubectl describe po calico-node-gvnnc -n kube-system查看：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m14s default-scheduler Successfully assigned kube-system/calico-node-gvnnc to k8s-master
Normal Pulling 2m14s kubelet, k8s-master Pulling image "calico/cni:v3.9.5"
Normal Pulled 99s kubelet, k8s-master Successfully pulled image "calico/cni:v3.9.5"
Normal Created 98s kubelet, k8s-master Created container upgrade-ipam
Normal Started 98s kubelet, k8s-master Started container upgrade-ipam
Normal Pulled 97s kubelet, k8s-master Container image "calico/cni:v3.9.5" already present on machine
Normal Created 97s kubelet, k8s-master Created container install-cni
Normal Started 97s kubelet, k8s-master Started container install-cni
Normal Pulling 96s kubelet, k8s-master Pulling image "calico/pod2daemon-flexvol:v3.9.5"
Normal Pulled 74s kubelet, k8s-master Successfully pulled image "calico/pod2daemon-flexvol:v3.9.5"
Normal Created 74s kubelet, k8s-master Created container flexvol-driver
Normal Started 74s kubelet, k8s-master Started container flexvol-driver
Normal Pulling 74s kubelet, k8s-master Pulling image "calico/node:v3.9.5"
Normal Pulled 47s kubelet, k8s-master Successfully pulled image "calico/node:v3.9.5"
Normal Created 47s kubelet, k8s-master Created container calico-node
Normal Started 47s kubelet, k8s-master Started container calico-node
安装Dashboard
部署Dashboard
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
kubectl apply -f recommended.yaml
部署完毕后, 执行kubectl get pods --all-namespaces查看pods状态，过几分钟全部变成Running
[root@ecs-babc-0004 kubernetes]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-5fc5dbfc47-4gvcb 1/1 Running 0 5m15s
kube-system calico-node-gvnnc 1/1 Running 0 5m15s
kube-system coredns-546565776c-sx5rj 1/1 Running 0 11m
kube-system coredns-546565776c-xtb92 1/1 Running 0 11m
kube-system etcd-k8s-master 1/1 Running 0 11m
kube-system kube-apiserver-k8s-master 1/1 Running 0 11m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 11m
kube-system kube-proxy-7jk4h 1/1 Running 0 11m
kube-system kube-scheduler-k8s-master 1/1 Running 0 11m
kubernetes-dashboard dashboard-metrics-scraper-66b49655d4-bb6px 0/1 ContainerCreating 0 8s
kubernetes-dashboard kubernetes-dashboard-74b4487bfc-5dw84 0/1 ContainerCreating 0 8s
创建用户
创建一个用于登录Dashboard的用户， 创建文件dashboard-adminuser.yaml内容如下：
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
执行命令kubectl apply -f dashboard-adminuser.yaml
生成证书
官方文档中提供了登录1.7.X以上版本的登录方式，但并不清晰， 半支烟没有完全按照该文档的方式进行操作。进入到.kube目录中执行3个命令：
[root@ecs-babc-0004 .kube]# grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt
[root@ecs-babc-0004 .kube]# grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key
[root@ecs-babc-0004 .kube]# openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"
第三条命令生成证书时会提示输入密码，可以直接两次回车跳过，然后再查看文件列表，看到多了3个证书相关文件：
kubecfg.p12即需要导入客户端机器的证书，将证书拷贝到客户端机器上，导入即可，半支烟的电脑是MAC，直接在~目录下执行以下命令即可，（xxx.xxx.xxx.xxx，是服务器的外网ip）
scp root@xxx.xxx.xxx.xxx:/root/.kube/kubecfg.p12 ./
MAC电脑，下载完p12文件，需要双击安装证书。安装证书时输入证书密码即可。
此时我们可以登录面板了， 访问地址： https://{k8s-master-ip}:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login，登录时会提示选择证书，确认后会提示输入当前用户名密码(注意是电脑的用户名密码)。此处的{k8s-master-ip}是指服务器的外网ip。
然后会弹出dashboard的登录界面：
登录Dashboard
在服务器上的.kube目录下执行kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')，获取Token。
[root@ecs-babc-0004 .kube]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Name: admin-user-token-9gjgz
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: a70bac13-dc07-49d2-9f4d-4296654ad66f
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1025 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWRoaGtiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiMjBkMTE0My1jZTk0LTQzNzktOWUxNC04ZjgwZjA2ZDg0NzkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.f6IbPGwIdFZWStzBj8_vmF01oWW5ccaCpPuVQNLSK1pgEqn0kNVK_x0RYSuKEnujObzpQQdFiRYcI6ITHja2PIVc5Nv83VCn5IaLvZdYuGZWUYRw0efJUBMA4J4N8-pRkiw6fYAuWLeGYghLNXL_nDdC_JkG75ASqrr3U1MVaikOcfrEPaI-T_AJ3TMYhI8aFoKiERpumu5W1K6Jl80Am9pWDX0Ywis5SSUP1VYfu-coI48EXSptcaxEyv58PrHUd6t_oMVV9rpqSxrNtMZvMeXqe8Hnl21vR7ls5yTZegYtHXSc3PKvCaIalKhYXAuhogNcIXHaMzvLSbf-DSQkVw
复制该Token到登录页，点击登录即可， 主界面如下：
添加Worker
重复执行 安装Docker ~ 安装Kubernetes-修改网络配置 的全部操作，初始化一个Worker机器。
执行如下命令将Worker加入集群：
kubeadm join 192.168.0.237:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:964c7b0c007ce17c979e631da17ad047dfa3bad76e407b6ee76d729ecf3cd9c7
- 注意: 此处的秘钥是初始化Master后生成的, 参考前文.
添加完毕后， 在Master上查看Worker节点状态：
[root@ecs-babc-0004 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
ecs-babc-0006 Ready <none> 93s v1.18.0
k8s-master Ready master 174m v1.18.0
当然在Dashboard也可查看:
总结
至此，在Centos上搭建Docker 和 Kubernetes已全部完成，步骤较多，只要有耐心，肯定能安装完毕。要知道半支烟在安装的时候，前后经历了十几次的重装系统，几乎崩溃，哈哈。
如果遇到问题也欢迎跟半支烟交流（关于我），感谢你的阅读。
参考文献
感谢以下文献作者：
错误链接：
标题：Centos搭建Docker和Kubernetes(一)
作者：程序员半支烟
地址：http://mangod.top/articles/2023/03/15/1678845388484.html