枯木谈自定义 Kubernetes CRD

原创 opskumu 云栈技术CSTC

01 理论概要

我们知道 Kubernetes 提供 CRD 机制用以扩展资源，在某些场景下我们可以利用 CRD 扩展通过 Kubernetes 存储资源，也就是把 Kubernetes 当作存储来使用。按照官方的说法 CRD + Controller = Operator，kubebuilder 可以生成这样的代码框架，并且可自定义选择是否生成 Controller。client 相关代码则可以通过 code-generator 生成，生成 informer、lister、clientset 等系列代码。通过这种方式，我们只需要在生成的代码中填充业务代码，而无需关心框架之外的代码，这样可以极大的提升开发效率和降低 CRD 的上手门槛。本文通过引入 ConfigMapHistory 这个 CRD 为例，带你从零到一构建和部署 CRD。

02 kubebuilder安装

kubebuilder 安装文档可以参考官档 Install kubebuilder

os=$(go env GOOS)arch=$(go env GOARCH)
# download kubebuilder and extract it to tmpcurl -L https://go.kubebuilder.io/dl/2.3.1/${os}/${arch} | tar -xz -C /tmp/
# move to a long-term location and put it on your path# (you'll need to set the KUBEBUILDER_ASSETS env var if you put it somewhere else)sudo mv /tmp/kubebuilder_2.3.1_${os}_${arch} /usr/local/kubebuilderexport PATH=$PATH:/usr/local/kubebuilder/bin

03 kubebuilder初始化

kubebuilder 代码初始化：

# github.com/opskumu 以实际仓库为主，本实例为 crd-examplemkdir -p $GOPATH/src/github.com/opskumu/crd-examplecd $GOPATH/src/github.com/opskumu/crd-example# 域名以实际域名为主，本实例为 opskumu.comkubebuilder init --domain opskumu.comkubebuilder edit --multigroup=true

查看当前生成代码的目录结构：

# tree -L 2 ./.├── Dockerfile├── Makefile├── PROJECT├── bin│   └── manager├── config│   ├── certmanager│   ├── default│   ├── manager│   ├── prometheus│   ├── rbac│   └── webhook├── go.mod├── go.sum├── hack│   └── boilerplate.go.txt└── main.go

04 kubebuilder创建API

# kubebuilder create api --group test --version v1 --kind ConfigMapHistoryCreate Resource [y/n]    # 创建 ConfigMapHistory 资源，此处选择 yyCreate Controller [y/n]  # 创建控制器，此处不需要，选择 nn# tree apis/test/v1   # apis 目录存放 group test v1 相关的资源 APIapis/test/v1├── configmaphistory_types.go├── groupversion_info.go└── zz_generated.deepcopy.go

05 自定义代码生成client代码

默认生成apis/test/v1/configmaphistory_types.go代码如下:

......// +kubebuilder:object:root=true
// ConfigMapHistory is the Schema for the configmaphistories APItype ConfigMapHistory struct {    metav1.TypeMeta   `json:",inline"`    metav1.ObjectMeta `json:"metadata,omitempty"`    Spec   ConfigMapHistorySpec   `json:"spec,omitempty"`    Status ConfigMapHistoryStatus `json:"status,omitempty"`}......

我们创建 ConfigMapHistory 类型的资源的目的是存放 ConfigMap 的历史记录，所以需要针对以上代码做一些修改：

......// +genclient// +kubebuilder:object:root=true
// ConfigMapHistory is the Schema for the configmaphistories APItype ConfigMapHistory struct {    metav1.TypeMeta   `json:",inline"`    metav1.ObjectMeta `json:"metadata,omitempty"`
    ConfigMap  string            `json:"configMap"`                // 关联 ConfigMap 名    Data       map[string]string `json:"data,omitempty"`           // 对应 ConfigMap 版本数据    BinaryData map[string]byte   `json:"binaryData,omitempty"`     // 对应 ConfigMap 版本二进制数据}......

注意 // +genclient 是额外添加的，为后续 code-generator 生成 client 代码打标签。types 文件修改之后，可以执行 make manifests 指令生成 CRD 文件：

# make manifests# ls config/crd/basestest.opskumu.com_configmaphistories.yaml

如果代码结构变更，需要再次执行 make manifest 以生成新的 CRD 文件创建 apis/test/v1/doc.go 、 apis/test/v1/register.go 、 hack/tools.go 和 hack/update-codegen.sh 文件

apis/test/v1/doc.go

/*Copyright 2017 The Kubernetes Authors.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.*/
// +k8s:deepcopy-gen=package// 注意 groupName 和实际相同// +groupName=test.opskumu.com
// Package v1 is the v1 version of the API.package v1

apis/test/v1/register.go

/*Copyright 2017 The Kubernetes Authors.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.*/
package v1
import (    "k8s.io/apimachinery/pkg/runtime/schema")
// SchemeGroupVersion is group version used to register these objects.var SchemeGroupVersion = GroupVersion
func Resource(resource string) schema.GroupResource {    return SchemeGroupVersion.WithResource(resource).GroupResource()}

hack/tools.go

// +build tools
/*Copyright 2019 The Kubernetes Authors.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.*/
// This package imports things required by build scripts, to force `go mod` to see them as dependenciespackage tools
import _ "k8s.io/code-generator"

tools.go 主要为了引入 k8s.io/code-generator 依赖。

hack/update-codegen.sh

#!/usr/bin/env bash
# Copyright 2017 The Kubernetes Authors.                                                                                       #                                                                                                                              # Licensed under the Apache License, Version 2.0 (the "License");                                                              # you may not use this file except in compliance with the License.                                                             # You may obtain a copy of the License at##     http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.
set -o errexitset -o nounsetset -o pipefail
SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..CODEGEN_PKG=${CODEGEN_PKG:-$(cd "${SCRIPT_ROOT}"; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ../code-generator)}
# generate the code with:# --output-base    because this script should also be able to run inside the vendor dir of#                  k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir#                  instead of the $GOPATH directly. For normal projects this can be dropped.bash "${CODEGEN_PKG}"/generate-groups.sh "client,informer,lister" \  github.com/opskumu/crd-example/generated github.com/opskumu/crd-example/apis \  test:v1 \  --output-base "$(dirname "${BASH_SOURCE[0]}")/../../../.." \  --go-header-file "${SCRIPT_ROOT}"/hack/boilerplate.go.txt
# To use your own boilerplate text append:#   --go-header-file "${SCRIPT_ROOT}"/hack/custom-boilerplate.go.txt

以上代码可以参考 sample-controller doc.go/register.go/tools.go/update-codegen.sh 内容，本文中 CRD 是 kubebuilder 生成的，因此内容稍有不同。

创建完成以上文件之后，执行以下命令生成代码：

# 注意此处版本以下几个包要一致，否则可能出现不兼容的情况# K8SVERSION=v0.19.6# go get -v k8s.io/code-generator@${K8SVERSION}# go get -v k8s.io/client-go@${K8SVERSION}# go get -v k8s.io/apimachinery@${K8SVERSION}# go mod vendor

把相关依赖包放入 vendor 中，以便 hack/update-codegen.sh 可以调用 code-generator 中的脚本。完成之后，执行生成代码指令：

# bash hack/update-codegen.shGenerating clientset for test:v1 at github.com/opskumu/crd-example/generated/clientsetGenerating listers for test:v1 at github.com/opskumu/crd-example/generated/listersGenerating informers for test:v1 at github.com/opskumu/crd-example/generated/informers# crd-example tree -L 1 generatedgenerated├── clientset├── informers└── listers

至此完成了添加 ConfigMapHistory CRD 和相关 client 代码的所有操作，如果需要生成其它 Kind 类型的重复以上操作即可（以上创建的文件不需要重复创建）。

06 创建CRD

kubectl create -f config/crd/bases/test.opskumu.com_configmaphistories.yaml

创建测试资源

# cat test.yamlapiVersion: "test.opskumu.com/v1"kind: ConfigMapHistorymetadata:  name: testconfigMap: "test"data:  test: test# kubectl create -f ./test.yamlconfigmaphistory.test.opskumu.com/test created# kubectl get configmaphistory.test.opskumu.comNAME   AGEtest   27s

完整示例代码参见 crd-example

参考文档：

sample-controller
Programming Kubernetes CRDs
混合kubebuilder与code generator编写CRD

长按二维码关注公众号

*本公众号所发布内容仅代表作者观点，不代表社区立场

继续滑动看下一个