k8s搭建基于containerd方式

早期 kubelet 创建容器工作原理

因为 docker 出生的比 k8s 早，所以 k8s 早期的容器运行时都是基于 docker 的，kubelet 通过 docker 的 api 创建容器。后来，k8s 官方不想绑死在 docker 这架马车上，就把容器运行时抽象出来，定义了一个接口，叫 CRI (container runtime interface)，容器运行时接口, 通过这个接口，kubelet 可以和任何容器运行时交互。但是，docker 并没有实现这个接口，k8s 也不想直接失去 docker 的用户，所以 k8s 官方在 kubelet 中实现了一个叫 docker-shim 的组件，这个组件简单来说就是把 cri 接口转换成 docker 的 api，这样 kubelet 就可以和 docker 交互了, 这个组件在 kuberbetes 1.24 版本中已经被移除了。至于实现了 cri 接口的容器运行时，比如 containerd，cri-o 等，kubelet 可以直接和它们交互。

docker、containerd、runc 的关系

因为 podman 等新兴 container runtime 的崛起，docker 不想失去定义标准的机会，所以 docker 官方把 containerd 从 docker 中分离出来，独立成一个项目，实现了 cri 接口，这种 kubelet 就可以通过 cri 直接调用 containerd 了。然后，docker 官方又把 runc 从 containerd 中分离出来，独立成一个项目，定义了一个叫 OCI (Open Container Initiative) 的标准，这个标准定义了容器的格式和运行时，runc 就是这个标准的实现，目前实现 oci 的还有 crun youki keta 等。

因为 containerd 和 runc 脱胎于 docker，docker 又不能维护两份代码，所以 docker 就通过调用 containerd ，containerd 再通过配置实现 oci 标准的 runc 来创建容器。当然，你也可以手动配置其他实现了 oci 标准的容器运行时。

crictl 与 kubelet ---CRI---> containerd ---exec---> containerd-shim-runc ---exec---> runc

准备

系统版本：ubuntu 20.04
k8s 版本：1.28.2-00

主机名	IP
master	192.168.168.141
node1	192.168.168.151
node2	192.168.168.152

分别配置节点的 /etc/hostname 和 /etc/hosts

master 节点：

root@master:~# cat /etc/hostname
master
root@master:~# cat /etc/hosts
127.0.0.1 localhost
192.168.168.141 master

node1 节点：

root@master:~# cat /etc/hostname
master
root@master:~# cat /etc/hosts
127.0.0.1 localhost
192.168.168.151 node1

node2 节点：

root@master:~# cat /etc/hostname
master
root@master:~# cat /etc/hosts
127.0.0.1 localhost
192.168.168.152 node2

一、三个节点安装 containerd

1.修改 apt 源

root@master:~# vim /etc/apt/sources.list
root@master:~# cat /etc/apt/sources.list
deb http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse

# deb http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse
# deb-src http://mirrors.aliyun.com/ubuntu/ focal-proposed main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse

更新 apt 源

root@master:~# apt update

2.安装 containerd

root@master:~# apt install containerd -y

查看版本信息

root@master:~# containerd --version
containerd github.com/containerd/containerd 1.7.2 
root@master:~# ctr version
Client:
  Version:  1.7.2
  Revision: 
  Go version: go1.20.3

Server:
  Version:  1.7.2
  Revision: 
  UUID: 20991b0e-900b-4d32-a8f0-fce9a0cbe3c1

查看服务状态

root@master:~# systemctl status containerd

二、三个节点安装 Kubernetes

1.增加 apt 源

root@master:~# apt install apt-transport-https ca-certificates curl -y
root@master:~# curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -

root@master:~# vim /etc/apt/sources.list.d/kubernetes.list
root@master:~# cat /etc/apt/sources.list.d/kubernetes.list 
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main

更新 apt 源

root@master:~# apt update

2.安装 kubelet、kubeadm、kubectl

kubeadm：用来初始化集群的指令
kubelet：在集群中的每个节点上用来启动 Pod 和容器等
kubectl：用来与集群通信的命令行工具

kubeadm 不能帮你安装或者管理 kubelet 或 kubectl，所以你需要确保它们与通过 kubeadm 安装的控制平面的版本相匹配。如果不这样做，则存在发生版本偏差的风险，可能会导致一些预料之外的错误和问题。然而，控制平面与 kubelet 间的相差一个次要版本不一致是支持的，但 kubelet 的版本不可以超过 API 服务器的版本。例如，1.7.0 版本的 kubelet 可以完全兼容 1.8.0 版本的 API 服务器，反之则不可以。

root@master:~# apt install kubelet=1.28.2-00 kubeadm=1.28.2-00 kubectl=1.28.2-00 -y

锁定这三个软件的版本，避免意外升级导致版本错误

root@master:~# apt-mark hold kubeadm kubelet kubectl

3.验证

root@master:~# crictl images
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
E0503 20:42:29.127164    4120 remote_image.go:119] "ListImages with filter from image service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\"" filter="&ImageFilter{Image:&ImageSpec{Image:,Annotations:map[string]string{},},}"
FATA[0000] listing images: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"

执行 crictl images，若出现如上错误 rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"

进行配置 crictl 后，再执行 crictl images

root@master:~# crictl config runtime-endpoint unix:///run/containerd/containerd.sock
root@master:~# crictl config image-endpoint unix:///run/containerd/containerd.sock
root@master:~# cat /etc/crictl.yaml
runtime-endpoint: "unix:///run/containerd/containerd.sock"
image-endpoint: "unix:///run/containerd/containerd.sock"
timeout: 0
debug: false
pull-image-on-create: false
disable-pull-on-run: false
root@master:~# crictl images
IMAGE               TAG                 IMAGE ID            SIZE

三、初始化 master 节点

root@master:~# kubeadm init --kubernetes-version=1.28.2 --apiserver-advertise-address=192.168.168.141 --apiserver-bind-port=6443 --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///run/containerd/containerd.sock --v=5

注意，apiserver-advertise-address 需要使用本机上网卡的 ip，否则的话会导致 etcd 绑定 ip 失败启动不了，从而 apiserver 也启动不了

若出现如下错误，表示需要配置 /proc/sys/net/bridge/bridge-nf-call-iptables

[preflight] Some fatal errors occurred:
    [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
    [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

配置 /proc/sys/net/bridge/bridge-nf-call-iptables

root@master:~# modprobe br_netfilter
root@master:~# echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
root@master:~# echo 1 > /proc/sys/net/ipv4/ip_forward

执行 kubeadm reset 后，再重新执行 kubeadm init

root@master:~# kubeadm reset
root@master:~# kubeadm init --kubernetes-version=1.28.2 --apiserver-advertise-address=192.168.168.141 --apiserver-bind-port=6443 --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --cri-socket=unix:///run/containerd/containerd.sock --v=5
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.168.141:6443 --token kov0rv.cvkzf26gkcv36urx \
    --discovery-token-ca-cert-hash sha256:9ed0b9fcc6f2ca7a6bafe558499d42f6bfbb83412bfb10b018fe1556d1df35b4

看到初始化成功，根据提示，配置 kube 配置文件

root@master:~# mkdir -p $HOME/.kube
root@master:~# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
root@master:~# chown $(id -u):$(id -g) $HOME/.kube/config

查看 node 运行情况

root@master:~# kubectl get nodes
NAME     STATUS     ROLES           AGE     VERSION
master   NotReady   control-plane   4m13s   v1.28.2

看到 master 是 NotReady 的状态，这时看 master 的信息或查看服务日志，看到是 cni plugin not initialized：

root@master:~# kubectl describe node master
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 03 May 2024 20:57:00 +0800   Fri, 03 May 2024 20:51:38 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 03 May 2024 20:57:00 +0800   Fri, 03 May 2024 20:51:38 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 03 May 2024 20:57:00 +0800   Fri, 03 May 2024 20:51:38 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 03 May 2024 20:57:00 +0800   Fri, 03 May 2024 20:51:38 +0800   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

root@master:~# journalctl -xeu kubelet
May 03 20:58:15 master kubelet[6571]: E0503 20:58:15.865775    6571 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin re>

配置 flannel.yml

root@master:~# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

master 出现了 Ready

root@master:~# kubectl get nodes
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   10m   v1.28.2

四、加入 master 节点

root@node1:~# kubeadm join 192.168.168.141:6443 --token kov0rv.cvkzf26gkcv36urx --discovery-token-ca-cert-hash sha256:9ed0b9fcc6f2ca7a6bafe558499d42f6bfbb83412bfb10b018fe1556d1df35b4

若出现如下错误，表示需要配置 /proc/sys/net/bridge/bridge-nf-call-iptables

[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
    [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

配置 /proc/sys/net/bridge/bridge-nf-call-iptables 后，重新执行 kubeadm join

root@node1:~# modprobe br_netfilter
root@node1:~# echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
root@node1:~# echo 1 > /proc/sys/net/ipv4/ip_forward

在 master 节点上，看 node 有没有加入成功

root@master:~# kubectl get nodes
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   40m   v1.28.2
node1    Ready    <none>          14m   v1.28.2
node2    Ready    <none>          47s   v1.28.2

五、部署应用

1.创建 yaml

nginx.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: dev
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: dev
  name: nginx-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-label
  template:
    metadata:
      labels:
        app: nginx-label
    spec:
      containers:
      - name: nginx
        image: nginx:1.23.3
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  namespace: dev
  name: nginx-service
  labels:
    app: nginx-label
spec:
  type: NodePort
  selector:
    app: nginx-label
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30001

2.部署

root@master:~# kubectl apply -f nginx.yaml 
namespace/dev created
deployment.apps/nginx-deployment created
service/nginx-service created

查看 namespace

root@master:~# kubectl get ns
NAME              STATUS   AGE
default           Active   3h52m
dev               Active   7m5s
kube-flannel      Active   3h41m
kube-node-lease   Active   3h52m
kube-public       Active   3h52m
kube-system       Active   3h52m

查看 pod

root@master:~# kubectl get pods -n dev
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-5677bf467b-2f6sc   1/1     Running   0          8m36s
nginx-deployment-5677bf467b-j5kv8   1/1     Running   0          8m36s
nginx-deployment-5677bf467b-lmxz6   1/1     Running   0          8m36s
root@master:~# kubectl get pods -o wide -A
NAMESPACE      NAME                                READY   STATUS    RESTARTS   AGE     IP                NODE     NOMINATED NODE   READINESS GATES
dev            nginx-deployment-5677bf467b-2f6sc   1/1     Running   0          8m5s    10.244.1.3        node1    <none>           <none>
dev            nginx-deployment-5677bf467b-j5kv8   1/1     Running   0          8m5s    10.244.1.2        node1    <none>           <none>
dev            nginx-deployment-5677bf467b-lmxz6   1/1     Running   0          8m5s    10.244.2.2        node2    <none>           <none>
kube-flannel   kube-flannel-ds-6h8rm               1/1     Running   0          3h12m   192.168.168.152   node2    <none>           <none>
kube-flannel   kube-flannel-ds-95zw8               1/1     Running   0          3h42m   192.168.168.141   master   <none>           <none>
kube-flannel   kube-flannel-ds-qfld7               1/1     Running   0          3h26m   192.168.168.151   node1    <none>           <none>
kube-system    coredns-6554b8b87f-672v5            1/1     Running   0          3h52m   10.244.0.2        master   <none>           <none>
kube-system    coredns-6554b8b87f-rzhx6            1/1     Running   0          3h52m   10.244.0.3        master   <none>           <none>
kube-system    etcd-master                         1/1     Running   0          3h53m   192.168.168.141   master   <none>           <none>
kube-system    kube-apiserver-master               1/1     Running   0          3h53m   192.168.168.141   master   <none>           <none>
kube-system    kube-controller-manager-master      1/1     Running   0          3h53m   192.168.168.141   master   <none>           <none>
kube-system    kube-proxy-9bqtr                    1/1     Running   0          3h26m   192.168.168.151   node1    <none>           <none>
kube-system    kube-proxy-hhjvl                    1/1     Running   0          3h52m   192.168.168.141   master   <none>           <none>
kube-system    kube-proxy-p9kzx                    1/1     Running   0          3h12m   192.168.168.152   node2    <none>           <none>
kube-system    kube-scheduler-master               1/1     Running   0          3h53m   192.168.168.141   master   <none>           <none>

查看 deployment

root@master:~# kubectl get deployments -o wide -A
NAMESPACE     NAME               READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                                SELECTOR
dev           nginx-deployment   3/3     3            3           14m     nginx        nginx:1.23.3                                                          app=nginx-label
kube-system   coredns            2/2     2            2           3h59m   coredns      registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1   k8s-app=kube-dns

查看 service

root@master:~# kubectl get services -o wide -A
NAMESPACE     NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default       kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP                  3h54m   <none>
dev           nginx-service   NodePort    10.103.139.98   <none>        80:30001/TCP             9m41s   app=nginx-label
kube-system   kube-dns        ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   3h54m   k8s-app=kube-dns

在 node1 和 node2 可以查看到具体的容器

root@node1:~# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD
f54e1b7da7b07       ac232364af842       10 minutes ago      Running             nginx               0                   7d347c306c628       nginx-deployment-5677bf467b-2f6sc
6255bd6ec8fea       ac232364af842       10 minutes ago      Running             nginx               0                   527e00fb92525       nginx-deployment-5677bf467b-j5kv8
2bdccc0dd06cc       1575deaad3b05       3 hours ago         Running             kube-flannel        0                   5f4e7a3c778a8       kube-flannel-ds-qfld7
4aaf5088f24e5       c120fed2beb84       3 hours ago         Running             kube-proxy          0                   1f8764fa99f22       kube-proxy-9bqtr

3.访问

root@master:~# curl http://10.244.1.2:80 
root@master:~# curl -L http://192.168.168.141:30001

六、crictl

crictl 是 k8s 的命令，ctr 是 containerd 的命令

crictl pull 的镜像实际上是在 k8s.io namespace 下，可以使用 ctr -n k8s.io images ls 查看

crictl pull docker.io/library/redis:alpine3.13

root@master:~# crictl images
IMAGE                                                                         TAG                 IMAGE ID            SIZE
docker.io/flannel/flannel-cni-plugin                                          v1.4.1-flannel1     1e3c860c213d6       4.71MB
docker.io/flannel/flannel                                                     v0.25.1             1575deaad3b05       30.7MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns                   v1.10.1             ead0a4a53df89       16.2MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd                      3.5.9-0             73deb9a3f7025       103MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver            v1.28.2             cdcab12b2dd16       34.7MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager   v1.28.2             55f13c92defb1       33.4MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy                v1.28.2             c120fed2beb84       24.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler            v1.28.2             7a5d9d67a13f6       18.8MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause                     3.9                 e6f1816883972       322kB
registry.k8s.io/pause                                                         3.8                 4873874c08efc       311kB

root@master:~# ctr -n k8s.io images ls -q
docker.io/flannel/flannel-cni-plugin:v1.4.1-flannel1
docker.io/flannel/flannel-cni-plugin@sha256:e88c0d84fa89679eb6cb6a28bc257d652ced8d1b2e44d54a592f0a2cd85dba53
docker.io/flannel/flannel:v0.25.1
docker.io/flannel/flannel@sha256:0483b9a62c8190ac36d9aa4f7f9b0d5f914353f7ef3b56727d1947b209e49e39
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.10.1
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns@sha256:90d3eeb2e2108a14fe2ecbef1bc1b5607834335d99c842a377f338aade9da028
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.9-0
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd@sha256:b124583790d2407fa140c01f42166e3292cc8191ef5d37034fe5a89032081b90
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.28.2
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver@sha256:6f3bb53840b8f48485635574360b170605174c177295bec6c8312eb3f5703d70
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.28.2
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager@sha256:83f6ba38e8eaac636a4a887bc1b79d7466226d7733a75edb268d85b8caf005a9
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.28.2
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy@sha256:2bd3090ff89e82dcd2b5e77927f996efa928b923cc0c0cdea4ccad35931073ea
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.28.2
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler@sha256:db4034d052464008e88c8c6fe96beeacbb04ebf52c867a0b5038af71ccce21d5
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
registry.cn-hangzhou.aliyuncs.com/google_containers/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097
registry.k8s.io/pause:3.8

近期文章

近期评论

java · 2024-05-03 0

k8s搭建基于containerd方式

早期 kubelet 创建容器工作原理

docker、containerd、runc 的关系

准备

一、三个节点安装 containerd

1.修改 apt 源

2.安装 containerd

二、三个节点安装 Kubernetes

1.增加 apt 源

2.安装 kubelet、kubeadm、kubectl

3.验证

三、初始化 master 节点

四、加入 master 节点

五、部署应用

1.创建 yaml

2.部署

3.访问

六、crictl

您可能还喜欢...

发表回复取消回复

java · 2024-05-03 0

早期 kubelet 创建容器工作原理

docker、containerd、runc 的关系

准备

一、三个节点安装 containerd

1.修改 apt 源

2.安装 containerd

二、三个节点安装 Kubernetes

1.增加 apt 源

2.安装 kubelet、kubeadm、kubectl

3.验证

三、初始化 master 节点

四、加入 master 节点

五、部署应用

1.创建 yaml

2.部署

3.访问

六、crictl

您可能还喜欢...

gcc的编译及动态静态库的使用

Java设计模式-桥接模式

jvm 相关命令(jinfo、jmap、jstat、jcmd、jstack)

发表回复 取消回复

发表回复取消回复