kubernetes上部署Dragonfly

Dragonfly可以很好的节省kubernetes集群下载镜像的带宽，下文记录了在一个已经配置好的kubernetes集群中如何部署Dragonfly。

Dragonfly分为两部分，均部署到namespace kube-system中：

• supernode，部署在master节点
• df-daemon，部署在所有节点

部署supernode

由于supernode需要在 /home/admin/dragonfly 目录下缓存镜像blob，不宜与根分区共享存储（万一Dragonfly的GC做的不好，会写满根分区，致命），所以我单独给虚拟机加了块盘，挂载到了 /data 目录下，通过hostpath挂载给supernode使用。

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: supernode
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: supernode
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: "node-role.kubernetes.io/master"
                  operator: Exists
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      containers:
      - name: supernode
        image: silenceshell/supernode:0.2.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8001
        - containerPort: 8002
        volumeMounts:
        - mountPath: /home/admin/supernode
          name: data
      volumes:
      - name: data
        hostPath:
          path: /data/dragonfly
          type: Directory
---
apiVersion: v1
kind: Service
metadata:
  name: supernode
  namespace: kube-system
  labels:
    svc: supernode
spec:
  ports:
  - port: 8001
    name: port1
    protocol: TCP
    targetPort: 8001
  - port: 8002
    name: port2
    protocol: TCP
    targetPort: 8002
  selector:
    app: supernode
  sessionAffinity: None

df-daemon通过svc supernode建立与supernode的连接。

通过kubectl创建后，有如下Pod。

$ kubectl get pods -n kube-system -o wide|grep super
supernode-6556bf8bf6-tj7d9         1/1     Running   0          18m   10.244.0.144    ubuntu-1   <none>

部署df-daemon

Daemonset很适合用来部署df-daemon。

df-daemon需要配置为hostnet模式，因为docker daemon需要连接df-daemon的65001端口。我的df-daemon的环境变量 REGISTRY 配置为 https://registry.docker-cn.com ，如果你是用来代理私有镜像，环境变量 REGISTRY 需要修改为对应的值。

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    k8s-app: df-daemon
  name: df-daemon
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: df-daemon
  template:
    metadata:
      labels:
        k8s-app: df-daemon
    spec:
      tolerations:
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"
      - key: "external"
        operator: "Equal"
        value: "ingress"
        effect: "NoSchedule"
      containers:
      - env:
        - name: REGISTRY
          value: https://registry.docker-cn.com
        image: silenceshell/df-daemon:0.2.0
        imagePullPolicy: Always
        name: df-daemon
        volumeMounts:
        - mountPath: /etc/dragonfly.conf
          name: config
          readOnly: true
          subPath: dragonfly.conf
        - mountPath: /root/.small-dragonfly/
          name: data
      dnsPolicy: ClusterFirstWithHostNet
      hostNetwork: true
      restartPolicy: Always
      volumes:
      - configMap:
          defaultMode: 384
          name: df-daemon-config
        name: config
      - hostPath:
          path: /data/dfdaemon
        name: data
---
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    k8s-app: df-daemon
  name: df-daemon-config
  namespace: kube-system
data:
  dragonfly.conf: |-
    [node]
    address=supernode

通过kubectl部署后，有如下Pod。可以看到，容器运行在宿主机网络中。

$ kubectl get pods -n kube-system -o wide|grep df-daemon
df-daemon-75rxn                    1/1     Running   1          20m    192.168.0.3   ubuntu-3   <none>
df-daemon-b25qt                    1/1     Running   0          20m    192.168.0.2   ubuntu-2   <none>
df-daemon-nkxsz                    1/1     Running   1          20m    192.168.0.1   ubuntu-1   <none>

修改Docker daemon配置

Dragonfly配置好后，就可以修改Docker daemon的配置了。

修改 /etc/docker/daemon.json ，将df-daemon配置为Docker daemon的mirror。

{
  "registry-mirrors": ["http://127.0.0.1:65001"]
}

重启 Docker daemon，通过 docker info 可以查看到如下信息：

Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 http://127.0.0.1:65001

至此，配置完成。