Kubernetes 二进制安装过程
以下文章来源于青木码林 ，作者qingmg
青木码林
.
工作的一些学习记录。
前置规划
六台装有 CentOS 的联网机器（ 本文使用 VM 生成 ）
三台装 master（ 设置 192.168.46.100 为 负债均衡 vip ） 三台装 worker
证书生成使用 cfssl 生成，习惯使用 openssl 的可以自行更换。
master
worker
软件版本
| 软件 | 版本 |
|---|---|
| keepalived（ https://keepalived.org/download.html ） | 2.2.1 |
| etcd（ https://github.com/etcd-io/etcd/releases/ ） | v3.4.14 |
| kube-apiserver，kube-controller-manager，kube-scheduler，kube-proxy | v1.21.0-alpha.2 |
| calico（https://docs.projectcalico.org/v3.17/manifests/calico.yaml） | v3.17 |
| coredns（https://github.com/coredns/deployment/blob/master/kubernetes/coredns.yaml.sed） | 1.8.0 |
服务器的前置工作
六台都需要执行
更新 hostname
三台 worker 必须执行，不能为同一个 hostname，不然后面 worker 节点上线会出错。
hostnamectl set-hostname xxx
更新 hosts 文件
cp /etc/hosts /etc/hosts.bak
cat >> /etc/hosts << EOF
192.168.46.101 k8s-m1
192.168.46.102 k8s-m2
192.168.46.103 k8s-m3
192.168.46.104 k8s-n1
192.168.46.105 k8s-n2
192.168.46.106 k8s-n3
EOF
关闭防火墙 和 selinux
临时方案
重启机器会失效
systemctl stop firewalld
setenforce 0
永久方案
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
cp /etc/selinux/config /etc/selinux/config.bak
sed -i 's/enforcing/disabled/' /etc/selinux/config
关闭交换分区（ 也就是 swap ）
临时方案
重启机器会失效
swapoff -a
永久方案
swapoff -a
cp /etc/fstab /etc/fstab.bak
sed -ri 's/.*swap.*/#&/' /etc/fstab
时间同步
用 chrony 同步
yum install -y chrony
systemctl start chronyd
chronyc sources
用 ntpdate 同步
yum install -y ntpdate
ntpdate time.windows.com
将桥接的 IPv4 流量传递到 iptables 的链
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
Master 机器部署进行时
临时路径为 /opt/work etcd 的工作路径为 /opt/etcd kubernetes 的工作路径为 /opt/kubernetes 大部分工作会在 k8s-m1 上执行，后将成品发送到其他机器上，通过第一行的注释判断在哪些机器上运行（ & 为且， | 为或 ）
部署 etcd 集群
下载 cfssl 工具并使其可执行
# ssh root@k8s-m1
cd /usr/local/bin
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O cfssl-certinfo
chmod +x cfssl*
生成 CA 证书
# ssh root@k8s-m1
mkdir -p /opt/work/etcd; cd /opt/work/etcd
cat > ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "k8s",
"OU": "system"
}
],
"ca": {
"expiry": "87600h"
}
}
EOF
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
ls ca*.pem
如果能看到 ca.pem 跟 ca-key.pem 则生成证书成功。
生成 etcd 的 CA 证书
# ssh root@k8s-m1
cd /opt/work/etcd
cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
cat > etcd-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.46.101",
"192.168.46.102",
"192.168.46.103"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "k8s",
"OU": "system"
}]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*.pem
如果能看到 etcd.pem 跟 etcd-key.pem 则生成证书成功。
部署
生成 etcd 需要的文件
启动参数参考：https://doczhcn.gitbook.io/etcd/index/index-1/configuration
# ssh root@k8s-m1
cd /opt/work/etcd
wget https://github.com/etcd-io/etcd/releases/download/v3.4.14/etcd-v3.4.14-linux-amd64.tar.gz
tar zxf etcd-v3.4.14-linux-amd64.tar.gz
mkdir -p /opt/etcd/{bin,cfg,data,ssl,shell}; cd /opt/etcd
cp /opt/work/etcd/etcd-v3.4.14-linux-amd64/etcd* bin/; chmod +x bin/etcd*
cp /opt/work/etcd/*.pem ssl/
cat > cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/opt/etcd/data"
ETCD_LISTEN_PEER_URLS="https://192.168.46.101:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.46.101:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.46.101:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.46.101:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.46.101:2380,etcd2=https://192.168.46.102:2380,etcd3=https://192.168.46.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
cat > etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=-/opt/etcd/cfg/etcd.conf
WorkingDirectory=/opt/etcd/
ExecStart=/usr/local/bin/etcd \\
--cert-file=/opt/etcd/ssl/etcd.pem \\
--key-file=/opt/etcd/ssl/etcd-key.pem \\
--trusted-ca-file=/opt/etcd/ssl/ca.pem \\
--peer-cert-file=/opt/etcd/ssl/etcd.pem \\
--peer-key-file=/opt/etcd/ssl/etcd-key.pem \\
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \\
--peer-client-cert-auth \\
--client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
cat > shell/init.sh << "EOF"
for i in $(ls /opt/etcd/bin/*); do ln -s $i /usr/local/bin/; done
ln -s /opt/etcd/etcd.service /usr/lib/systemd/system/
EOF
chmod +x shell/init.sh
拷贝到另外俩台 master 机器上
# ssh root@k8s-m1
scp -r /opt/etcd root@k8s-m2:/opt/scp -r /opt/etcd root@k8s-m3:/opt/
在 k8s-m2 和 k8s-m3 更改 etcd.conf 中的 ip 和 ETCD_NAME 值
# ssh root@k8s-m2 & ssh root@k8s-m3
vim /opt/etcd/cfg/etcd.conf
############################
#### 需要修改的
#ETCD_NAME
#ETCD_LISTEN_PEER_URLS
#ETCD_LISTEN_CLIENT_URLS
#ETCD_INITIAL_ADVERTISE_PEER_URLS
#ETCD_ADVERTISE_CLIENT_URLS
############################
执行初始化脚本完成初始化
# ssh root@k8s-m1 & ssh root@k8s-m2 & ssh root@k8s-m3
cd /opt/etcd/shell/
./init.sh
启动 etcd 并设置开机启动
# ssh root@k8s-m1 & ssh root@k8s-m2 & ssh root@k8s-m3
systemctl daemon-reload
systemctl start etcd
systemctl status etcd
systemctl enable etcd
查看 etcd 集群状态
# ssh root@k8s-m1 | ssh root@k8s-m2 | ssh root@k8s-m3
ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem --endpoints=https://192.168.46.101:2379,https://192.168.46.102:2379,https://192.168.46.103:2379 endpoint health
部署 master kubernetes
keepalived 安装部署
# ssh root@k8s-m1 & ssh root@k8s-m2 & ssh root@k8s-m3
yum install openssl-devel -y
mkdir /opt/keepalived; cd /opt/keepalived
wget https://keepalived.org/software/keepalived-2.2.1.tar.gz
tar zxf keepalived-2.2.1.tar.gz
cd keepalived-2.2.1/
./configure --prefix=$PWD/install
make && make install
ln -s /opt/keepalived/keepalived-2.2.1/install/sbin/keepalived /usr/sbin/
mkdir -p /etc/keepalived/
cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
vrrp_script chk_6443 {
script "/etc/keepalived/kubeapi_check.sh"
interval 2
}
vrrp_instance VI_1 {
state BACKUP # 状态参数 master/backup 只是说明
interface ens33 # 虚拟IP地址放置的网卡位置
virtual_router_id 100 # 同一家族要一直，同一个集群id一致
priority 100 # 优先级决定是主还是备 越大越优先
nopreempt
advert_int 1 # 主备通讯时间间隔
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.46.100
}
track_script {
chk_6443
}
}
EOF
cat > /etc/keepalived/kubeapi_check.sh << "EOF"
#!/bin/bash
ALIVE=`ps -ef|grep kube-apiserver|grep -v grep|awk '{print $2}'`
if [ "$ALIVE" ]; then
exit 0
else
exit 1
fi
EOF
chmod +x /etc/keepalived/kubeapi_check.sh
######################################################
####修改 k8s-m2、k8s-m3 /etc/keepalived/keepalived.conf 中的 priority 值，三台机器不相等即可
######################################################
systemctl start keepalived
systemctl status keepalived
systemctl enable keepalived
安装部署 kube-apiserver、kube-controller-manager、kube-scheduler
拷贝需要的 bin 文件
# ssh root@k8s-m1
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs,shell}
mkdir /opt/work/kubernetes; cd /opt/work/kubernetes
cp /opt/work/etcd/ca*.pem .
cp /opt/work/etcd/ca-config.json .
wget https://dl.k8s.io/v1.21.0-alpha.2/kubernetes-server-linux-amd64.tar.gz
tar zxf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/
cp kube-apiserver kube-controller-manager kube-scheduler kubectl /opt/kubernetes/bin/
生成 kube-apiserver 需要的文件
启动参数参考：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-apiserver/
# ssh root@k8s-m1
cd /opt/work/kubernetes
cat > kube-apiserver-csr.json << EOF
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"192.168.46.100",
"192.168.46.101",
"192.168.46.102",
"192.168.46.103",
"192.168.46.104",
"192.168.46.105",
"192.168.46.106",
"10.255.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "k8s",
"OU": "system"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
cat > token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
cat > kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
--anonymous-auth=false \\
--bind-address=0.0.0.0 \\
--secure-port=6443 \\
--advertise-address=0.0.0.0 \\
--insecure-port=0 \\
--authorization-mode=Node,RBAC \\
--runtime-config=api/all=true \\
--enable-bootstrap-token-auth \\
--service-cluster-ip-range=10.255.0.0/16 \\
--token-auth-file=/opt/kubernetes/token.csv \\
--service-node-port-range=30000-50000 \\
--tls-cert-file=/opt/kubernetes/ssl/kube-apiserver.pem \\
--tls-private-key-file=/opt/kubernetes/ssl/kube-apiserver-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--kubelet-client-certificate=/opt/kubernetes/ssl/kube-apiserver.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/kube-apiserver-key.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/etcd.pem \\
--etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \\
--etcd-servers=https://192.168.46.101:2379,https://192.168.46.102:2379,https://192.168.46.103:2379 \\
--enable-swagger-ui=true \\
--allow-privileged=true \\
--apiserver-count=3 \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/kube-apiserver-audit.log \\
--event-ttl=1h \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/opt/kubernetes/logs \\
--v=4"
EOF
cat > kube-apiserver.service << "EOF"
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
cd /opt/kubernetes
cp /opt/work/kubernetes/*.pem ssl/
cp /opt/work/kubernetes/kube-apiserver.conf cfg/
cp /opt/work/kubernetes/kube-apiserver.service /opt/work/kubernetes/token.csv .
生成 kubectl 需要的文件
# ssh root@k8s-m1
cd /opt/work/kubernetes
cat > kubectl-csr.json << EOF
{
"CN": "kubectl",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "system:masters",
"OU": "system"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kubectl-csr.json | cfssljson -bare kubectl
cp kubectl*.pem /opt/kubernetes/ssl/
生成 kube-controller-manager 需要的文件
启动参数参考：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-controller-manager/
# ssh root@k8s-m1
cd /opt/work/kubernetes
cat > kube-controller-manager-csr.json << EOF
{
"CN": "system:kube-controller-manager",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"192.168.46.101",
"192.168.46.102",
"192.168.46.103"
],
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "system:kube-controller-manager",
"OU": "system"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
cat > kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--bind-address=127.0.0.1 \\
--kubeconfig=/opt/kubernetes/kube-controller-manager.kubeconfig \\
--service-cluster-ip-range=10.255.0.0/16 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.0.0.0/16 \\
--experimental-cluster-signing-duration=87600h \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--leader-elect=true \\
--feature-gates=RotateKubeletServerCertificate=true \\
--controllers=*,bootstrapsigner,tokencleaner \\
--horizontal-pod-autoscaler-use-rest-clients=true \\
--horizontal-pod-autoscaler-sync-period=10s \\
--tls-cert-file=/opt/kubernetes/ssl/kube-controller-manager.pem \\
--tls-private-key-file=/opt/kubernetes/ssl/kube-controller-manager-key.pem \\
--use-service-account-credentials=true \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/opt/kubernetes/logs \\
--v=2"
EOF
cat > kube-controller-manager.service << "EOF"
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
cp kube-controller*.pem /opt/kubernetes/ssl/
cp kube-controller-manager.conf /opt/kubernetes/cfg/
cp kube-controller-manager.service /opt/kubernetes/
生成 kube-scheduler 需要的文件
启动参数参考：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-scheduler/
# ssh root@k8s-m1
cd /opt/work/kubernetes
cat > kube-scheduler-csr.json << EOF
{
"CN": "system:kube-scheduler",
"key": {
"algo": "rsa",
"size": 2048
},
"hosts": [
"127.0.0.1",
"192.168.46.101",
"192.168.46.102",
"192.168.46.103"
],
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "system:kube-scheduler",
"OU": "system"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
cat > kube-scheduler.conf << EOF
KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \\
--kubeconfig=/opt/kubernetes/kube-scheduler.kubeconfig \\
--leader-elect=true \\
--alsologtostderr=true \\
--logtostderr=false \\
--log-dir=/opt/kubernetes/logs \\
--v=2"
EOF
cat > kube-scheduler.service << "EOF"
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
cp kube-scheduler*.pem /opt/kubernetes/ssl/
cp kube-scheduler.conf /opt/kubernetes/cfg/
cp kube-scheduler.service /opt/kubernetes/
增加初始化脚本
# ssh root@k8s-m1
cd /opt/kubernetes
cat > shell/init.sh << "EOF"
for i in $(ls /opt/kubernetes/bin/*); do ln -s $i /usr/local/bin/; done
for i in $(ls /opt/kubernetes/*.service); do ln -s $i /usr/lib/systemd/system/; done
EOF
cat > shell/init-config.sh << "EOF"
kubectl config set-cluster kubernetes --certificate-authority=/opt/etcd/ssl/ca.pem --embed-certs=true --server=https://192.168.46.100:6443 --kubeconfig=kubectl.config
kubectl config set-credentials kubectl --client-certificate=/opt/kubernetes/ssl/kubectl.pem --client-key=/opt/kubernetes/ssl/kubectl-key.pem --embed-certs=true --kubeconfig=kubectl.config
kubectl config set-context kubernetes --cluster=kubernetes --user=kubectl --kubeconfig=kubectl.config
kubectl config use-context kubernetes --kubeconfig=kubectl.config
kubectl config set-cluster kubernetes --certificate-authority=/opt/etcd/ssl/ca.pem --embed-certs=true --server=https://192.168.46.100:6443 --kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager --client-certificate=/opt/kubernetes/ssl/kube-controller-manager.pem --client-key=/opt/kubernetes/ssl/kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-cluster kubernetes --certificate-authority=/opt/etcd/ssl/ca.pem --embed-certs=true --server=https://192.168.46.100:6443 --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler --client-certificate=/opt/kubernetes/ssl/kube-scheduler.pem --client-key=/opt/kubernetes/ssl/kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
EOF
chmod +x shell/*.sh
拷贝到另外俩台 master 机器上
# ssh root@k8s-m1
scp -r /opt/kubernetes root@k8s-m2:/opt/
scp -r /opt/kubernetes root@k8s-m3:/opt/
执行初始化脚本完成初始化
# ssh root@k8s-m1 & ssh root@k8s-m2 & ssh root@k8s-m3
cd /opt/kubernetes
./shell/init.sh
./shell/init-config.sh
启动并设置开机启动
# ssh root@k8s-m1 & ssh root@k8s-m2 & ssh root@k8s-m3
systemctl start kube-apiserver
systemctl start kube-controller-manager
systemctl start kube-scheduler
systemctl enable kube-apiserver
systemctl enable kube-controller-manager
systemctl enable kube-scheduler
部署 kubectl
# ssh root@k8s-m1 & ssh root@k8s-m2 & ssh root@k8s-m3
cd /opt/kubernetes
mkdir ~/.kube
cp kubectl.config ~/.kube/config
生成 clusterrolebinding 使 kubectl 可用
# ssh root@k8s-m1 | ssh root@k8s-m2 | ssh root@k8s-m3
kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
一些 kubectl 使用命令
可参考文档：https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands
kubectl cluster-info
kubectl get cs
实现 kubectl 命令用 tab键 可以补全
如果要开机就可以用，需要加到 /etc/bashrc 中
# ssh root@k8s-m1 & ssh root@k8s-m2 & ssh root@k8s-m3
source <(kubectl completion bash)
worker 机器部署进行时
kubernetes 的工作路径为 /opt/kubernetes 大部分工作会在 k8s-m1 上执行，后将成品发送到其他机器上，通过第一行的注释判断在哪些机器上运行（ & 为且， | 为或 ）
安装 Dokcer
三台 worker 机器上一起运行
使用阿里云源安装
# ssh root@k8s-n1 & ssh root@k8s-n2 & ssh root@k8s-n3
wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum install docker-ce -y
设置镜像加速
# ssh root@k8s-n1 & ssh root@k8s-n2 & ssh root@k8s-n3
mkdir -p /etc/docker
cat > /etc/docker/daemon.json << EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": [
"https://registry.docker-cn.com",
"https://docker.mirrors.ustc.edu.cn"
]
}
EOF
启动并设置开机启动
# ssh root@k8s-n1 & ssh root@k8s-n2 & ssh root@k8s-n3
systemctl start docker
systemctl enable docker
部署 worker kubernetes
回到 k8s-m1 上生成 worker 机器需要的文件
拷贝需要的 bin 文件
# ssh root@k8s-m1
mkdir -p /opt/kubernetes-node/{bin,cfg,ssl,shell,kubelet/logs,kube-proxy/logs}
cd /opt/work/kubernetes/kubernetes/server/bin
cp kubelet kube-proxy /opt/kubernetes-node/bin/
cp /opt/kubernetes/ssl/ca*.pem /opt/kubernetes-node/ssl/
生成 kubelet 需要的文件
启动参数参考：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kubelet/
# ssh root@k8s-m1
cd /opt/kubernetes-node
kubectl config set-cluster kubernetes --certificate-authority=/opt/kubernetes/ssl/ca.pem --embed-certs=true --server=https://192.168.46.100:6443 --kubeconfig=kubelet-bootstrap.kubeconfig
kubectl config set-credentials kubelet-bootstrap --token=$(awk -F "," '{print $1}' /opt/kubernetes/token.csv) --kubeconfig=kubelet-bootstrap.kubeconfig
kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
cat > cfg/kubelet-config.yaml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: systemd
clusterDNS:
- 10.255.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
x509:
clientCAFile: /opt/kubernetes/ssl/ca.pem
webhook:
enabled: true
cacheTTL: 2m0s
anonymous:
enabled: false
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
EOF
cat > kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service
[Service]
WorkingDirectory=/opt/kubernetes/kubelet
ExecStart=/usr/local/bin/kubelet \
--bootstrap-kubeconfig=/opt/kubernetes/kubelet-bootstrap.kubeconfig \
--cert-dir=/opt/kubernetes/ssl \
--kubeconfig=/opt/kubernetes/kubelet.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yaml \
--network-plugin=cni \
--pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/opt/kubernetes/kubelet/logs \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
生成 kube-proxy 需要的文件
启动参数参考：https://kubernetes.io/zh/docs/reference/command-line-tools-reference/kube-proxy/
# ssh root@k8s-m1
cd /opt/kubernetes-node
cp /opt/work/kubernetes/ca-config.json .
cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "GuangDong",
"L": "GuangZhou",
"O": "k8s",
"OU": "system"
}
]
}
EOF
cfssl gencert -ca=ssl/ca.pem -ca-key=ssl/ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
mv *.pem ssl/
kubectl config set-cluster kubernetes --certificate-authority=ssl/ca.pem --embed-certs=true --server=https://192.168.46.100:6443 --kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy --client-certificate=ssl/kube-proxy.pem --client-key=ssl/kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
cat > cfg/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.46.104
clientConnection:
kubeconfig: /opt/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 192.168.0.0/16
healthzBindAddress: 192.168.46.104:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.46.104:10249
mode: "ipvs"
EOF
cat > kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
WorkingDirectory=/opt/kubernetes/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
--config=/opt/kubernetes/cfg/kube-proxy.yaml \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/opt/kubernetes/kube-proxy/logs \
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
增加初始化脚本
# ssh root@k8s-m1
cd /opt/kubernetes-node
mv kube-pro*csr* /opt/work/kubernetes/
rm -f ca-config.json
cat > shell/init.sh << "EOF"
for i in $(ls /opt/kubernetes/bin/*);do ln -s $i /usr/local/bin/;done
for i in $(ls /opt/kubernetes/*.service);do ln -s $i /usr/lib/systemd/system/;done
EOF
chmod +x shell/init.sh
拷贝到三台 worker 机器上
# ssh root@k8s-m1
scp -r /opt/kubernetes-node root@k8s-n1:/opt/kubernetes
scp -r /opt/kubernetes-node root@k8s-n2:/opt/kubernetes
scp -r /opt/kubernetes-node root@k8s-n3:/opt/kubernetes
执行初始化脚本完成初始化
# ssh root@k8s-n1 & ssh root@k8s-n2 & ssh root@k8s-n3
cd /opt/kubernetes
./shell/init.sh
启动并设置开机启动
# ssh root@k8s-n1 & ssh root@k8s-n2 & ssh root@k8s-n3
cd /opt/kubernetes
#####################################################
########### xxxxx 是本机IP
#####################################################
sed -i 's/192.168.46.104/xxxxx/' cfg/kube-proxy.yaml
systemctl daemon-reload
systemctl start kubelet
systemctl start kube-proxy
systemctl status kubelet
systemctl status kube-proxy
systemctl enable kubelet
systemctl enable kube-proxy
查看并操作 worker 节点上线
回到 k8s-m1，通过执行 kubectl get csr 可以发现多了三条 condition=peding 的记录，同时执行 kubectl get nodes 查看已有的 worker 节点，如下图则为正常。
此时，执行命令 kubectl certificate approve ${NAME}，再执行上述俩命令查看结果
# ssh root@k8s-m1
kubectl get csr | awk '{print $1}' | grep -v NAME | xargs kubectl certificate approve
kubectl get csr
kubectl get nodes
结果如下为正常：
可以瞧见，上来了三个。
!!!!!!注：如果使用本文的配置，且三台 worker 的主机名（ hostname ）相同，会出现只有一条 node 记录的问题
部署网络组件使 worker 可用
可以自行抉择是想用 calico 、 flannel 、 canal 或者是其他的。 calico：https://github.com/projectcalico/cni-plugin flannel：https://github.com/coreos/flannel canal ：https://github.com/projectcalico/canal
本文使用 calico，获得 yaml 后执行 kubectl apply -f xx.yaml 就行
ssh root@k8s-m1
mkdir /opt/kubernetes/plugins; cd /opt/kubernetes/plugins
wget https://docs.projectcalico.org/v3.17/manifests/calico.yaml
kubectl apply -f calico.yaml
然后通过 kubectl get nodes 查看节点状态，通过 kubectl get pods -A 查看进度
静等一下，等 READY = 1/1 ， STATUS = RUNNING 即可
部署 coredns
ssh root@k8s-m1
cd /opt/kubernetes/plugins
wget https://github.com/coredns/deployment/blob/master/kubernetes/coredns.yaml.sed
cp coredns.yaml.sed coredns.yaml
sed -i 's/CLUSTER_DOMAIN REVERSE_CIDRS/cluster.local in-addr.arpa ip6.arpa/' coredns.yaml
sed -i 's/STUBDOMAINS//' coredns.yaml
sed -i 's/UPSTREAMNAMESERVER/\/etc\/resolv.conf/' coredns.yaml
sed -i 's/CLUSTER_DNS_IP/10.255.0.2/' coredns.yaml
kubectl apply -f coredns.yaml
通过命令 kubectl get pods -A 等 READY = 1/1 ， STATUS = RUNNING 便完成了。
由于可能下载不到，或者已经更新了导致不对，下面贴出本文现成的 coredns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . /etc/resolv.conf {
max_concurrent 1000
}
cache 30
loop
reload
loadbalance
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/name: "CoreDNS"
spec:
# replicas: not specified here:
# 1. Default is 1.
# 2. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
spec:
priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
nodeSelector:
kubernetes.io/os: linux
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: k8s-app
operator: In
values: ["kube-dns"]
topologyKey: kubernetes.io/hostname
containers:
- name: coredns
image: coredns/coredns:1.8.0
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.255.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- name: metrics
port: 9153
protocol: TCP
部署 nginx 应用试验结果
ssh root@k8s-m1
mkdir /opt/kubernetes/workspace; cd /opt/kubernetes/workspace
cat > nginx.yaml << EOF
---
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-controller
spec:
replicas: 2
selector:
name: nginx
template:
metadata:
labels:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.19.6
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-service-nodeport
spec:
ports:
- port: 80
targetPort: 80
nodePort: 30001
protocol: TCP
type: NodePort
selector:
name: nginx
EOF
kubectl apply -f nginx.yaml
查看进度的命令
ssh root@k8s-m1
kubectl get svc
kubectl get pods
通过访问 任一nodeIP:30001 ，访问到 nginx 主页即为安装成功。