今天使用kubebuilder，在kubernetes集群之外跑起了一个自定义controller，整理笔记如下。

目标：

1）学习使用kubebuilder创建工程，运行工程。

2）创建CDR(自定资源定义)，CR（自定资源）

3）修改CR（自定资源），自定义controller能感知。

环境说明：

1) 我已经搭建好K8S集群，在集群外有一台workstation机，今天的所有操作在workstation上完成。

2) workstation上已安装kubebuiler,go语言。

[root@workstation ~]# kubebuilder version
Version: main.version{KubeBuilderVersion:"3.3.0", KubernetesVendor:"1.23.1", GitCommit:"47859bf2ebf96a64db69a2f7074ffdec7f15c1ec", BuildDate:"2022-01-18T17:03:29Z", GoOs:"linux", GoArch:"amd64"}

 [root@workstation ~]# go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/go-path/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go/go-path"
GOPRIVATE=""
GOPROXY="https://goproxy.cn,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.7"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build550501954=/tmp/go-build -gno-record-gcc-switches"
[root@workstation ~]# 

3) workstation上已正确安装kubectl，并且~/.kube/config已从master主机上复制到workstation机器上。即，workstation已经要可以访问k8s上的资源：

[root@workstation ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:58:47Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
[root@workstation ~]# 

操作步骤：

第1步：执行：kubebuilder init --domain my.domain --repo my.domain/guestbook，

[root@workstation ~]# 
[root@workstation ~]# mkdir -p ~/projects/guestbook
[root@workstation ~]# cd ~/projects/guestbook
[root@workstation guestbook]# kubebuilder init --domain my.domain --repo my.domain/guestbook
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.11.0
Update dependencies:
$ go mod tidy
Next: define a resource with:
$ kubebuilder create api
[root@workstation guestbook]# 

第2步：执行：kubebuilder create api --group webapp --version v1 --kind Guestbook

其中, 当询问"Create Resource [y/n]"及Create Controller [y/n]时，都输入y.

这个过程要下载一些go lang的依赖，需要等一段时间。

[root@workstation guestbook]# kubebuilder create api --group webapp --version v1 --kind Guestbook
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing kustomize manifests for you to edit...
Writing scaffold for you to edit...
api/v1/guestbook_types.go
controllers/guestbook_controller.go
Update dependencies:
$ go mod tidy
Running make:
$ make generate
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/controller-tools/cmd/controller-gen@v0.8.0
go get: installing executables with 'go get' in module mode is deprecated.
	To adjust and download dependencies of the current module, use 'go get -d'.
	To install using requirements of the current module, use 'go install'.
	To install ignoring the current module, use 'go install' with a version,
	like 'go install example.com/cmd@latest'.
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.
go get: added github.com/fatih/color v1.12.0
go get: added github.com/go-logr/logr v1.2.0
go get: added github.com/gobuffalo/flect v0.2.3
go get: added github.com/gogo/protobuf v1.3.2
go get: added github.com/google/go-cmp v0.5.6
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.12
go get: added github.com/mattn/go-colorable v0.1.8
go get: added github.com/mattn/go-isatty v0.0.12
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.2
go get: added github.com/spf13/cobra v1.2.1
go get: added github.com/spf13/pflag v1.0.5
go get: added golang.org/x/mod v0.4.2
go get: added golang.org/x/net v0.0.0-20210825183410-e898025ed96a
go get: added golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e
go get: added golang.org/x/text v0.3.7
go get: added golang.org/x/tools v0.1.6-0.20210820212750-d4cc65f0b2ff
go get: added golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.4.0
go get: added gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
go get: added k8s.io/api v0.23.0
go get: added k8s.io/apiextensions-apiserver v0.23.0
go get: added k8s.io/apimachinery v0.23.0
go get: added k8s.io/klog/v2 v2.30.0
go get: added k8s.io/utils v0.0.0-20210930125809-cb0fa318a74b
go get: added sigs.k8s.io/controller-tools v0.8.0
go get: added sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6
go get: added sigs.k8s.io/structured-merge-diff/v4 v4.1.2
go get: added sigs.k8s.io/yaml v1.3.0
/root/projects/guestbook/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Next: implement your new API and generate the manifests (e.g. CRDs,CRs) with:
$ make manifests
[root@workstation guestbook]# 

第3步： 执行make install

这个过程要下载一些go lang的依赖，需要等一段时间。

 [root@workstation guestbook]# make install
/root/projects/guestbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
go get: installing executables with 'go get' in module mode is deprecated.
	To adjust and download dependencies of the current module, use 'go get -d'.
	To install using requirements of the current module, use 'go install'.
	To install ignoring the current module, use 'go install' with a version,
	like 'go install example.com/cmd@latest'.
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.
go get: added cloud.google.com/go v0.38.0
go get: added github.com/Azure/go-autorest/autorest v0.9.0
go get: added github.com/Azure/go-autorest/autorest/adal v0.5.0
go get: added github.com/Azure/go-autorest/autorest/date v0.1.0
go get: added github.com/Azure/go-autorest/logger v0.1.0
go get: added github.com/Azure/go-autorest/tracing v0.5.0
go get: added github.com/PuerkitoBio/purell v1.1.1
go get: added github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go get: added github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a
go get: added github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
go get: added github.com/davecgh/go-spew v1.1.1
go get: added github.com/dgrijalva/jwt-go v3.2.0+incompatible
go get: added github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633
go get: added github.com/evanphx/json-patch v4.9.0+incompatible
go get: added github.com/go-errors/errors v1.0.1
go get: added github.com/go-openapi/analysis v0.19.5
go get: added github.com/go-openapi/errors v0.19.2
go get: added github.com/go-openapi/jsonpointer v0.19.3
go get: added github.com/go-openapi/jsonreference v0.19.3
go get: added github.com/go-openapi/loads v0.19.4
go get: added github.com/go-openapi/runtime v0.19.4
go get: added github.com/go-openapi/spec v0.19.5
go get: added github.com/go-openapi/strfmt v0.19.5
go get: added github.com/go-openapi/swag v0.19.5
go get: added github.com/go-openapi/validate v0.19.8
go get: added github.com/go-stack/stack v1.8.0
go get: added github.com/gogo/protobuf v1.3.1
go get: added github.com/golang/protobuf v1.3.2
go get: added github.com/google/gofuzz v1.1.0
go get: added github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go get: added github.com/googleapis/gnostic v0.1.0
go get: added github.com/gophercloud/gophercloud v0.1.0
go get: added github.com/hashicorp/errwrap v1.0.0
go get: added github.com/hashicorp/go-cleanhttp v0.5.0
go get: added github.com/hashicorp/go-multierror v1.1.0
go get: added github.com/hashicorp/go-safetemp v1.0.0
go get: added github.com/hashicorp/go-version v1.1.0
go get: added github.com/inconshreveable/mousetrap v1.0.0
go get: added github.com/json-iterator/go v1.1.8
go get: added github.com/mailru/easyjson v0.7.0
go get: added github.com/mattn/go-runewidth v0.0.7
go get: added github.com/mitchellh/go-homedir v1.1.0
go get: added github.com/mitchellh/go-testing-interface v1.0.0
go get: added github.com/mitchellh/mapstructure v1.1.2
go get: added github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go get: added github.com/modern-go/reflect2 v1.0.1
go get: added github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00
go get: added github.com/olekukonko/tablewriter v0.0.4
go get: added github.com/pkg/errors v0.9.1
go get: added github.com/pmezard/go-difflib v1.0.0
go get: added github.com/qri-io/starlib v0.4.2-0.20200213133954-ff2e8cd5ef8d
go get: added github.com/spf13/cobra v1.0.0
go get: added github.com/spf13/pflag v1.0.5
go get: added github.com/stretchr/testify v1.6.1
go get: added github.com/ulikunitz/xz v0.5.5
go get: added github.com/xlab/treeprint v0.0.0-20181112141820-a009c3971eca
go get: added github.com/yujunz/go-getter v1.4.1-lite
go get: added go.mongodb.org/mongo-driver v1.1.2
go get: added go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5
go get: added golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go get: added golang.org/x/net v0.0.0-20200625001655-4c5254603344
go get: added golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
go get: added golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd
go get: added golang.org/x/text v0.3.2
go get: added golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go get: added google.golang.org/appengine v1.5.0
go get: added gopkg.in/inf.v0 v0.9.1
go get: added gopkg.in/yaml.v2 v2.3.0
go get: added gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go get: added k8s.io/api v0.18.10
go get: added k8s.io/apimachinery v0.18.10
go get: added k8s.io/client-go v0.18.10
go get: added k8s.io/klog v1.0.0
go get: added k8s.io/kube-openapi v0.0.0-20200410145947-61e04a5be9a6
go get: added k8s.io/utils v0.0.0-20200324210504-a9aa75ae1b89
go get: added sigs.k8s.io/kustomize/api v0.6.5
go get: added sigs.k8s.io/kustomize/cmd/config v0.8.5
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
go get: added sigs.k8s.io/kustomize/kyaml v0.9.4
go get: added sigs.k8s.io/structured-merge-diff/v3 v3.0.0
go get: added sigs.k8s.io/yaml v1.2.0
/root/projects/guestbook/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/guestbooks.webapp.my.domain configured
[root@workstation guestbook]# 

这一步会创建crd资源，查看一下：

[root@workstation guestbook]# kubectl get crd|grep guest
guestbooks.webapp.my.domain                           2022-03-31T07:30:08Z
[root@workstation guestbook]# 

第4步：修改controller.go，加入自己的业务逻辑。正常情况下，这里要加如何部署pod，但今天只是演示，所以只打印日志。修改如下：

 如果不修改的话，下面的操作，将看不到日志，无法知道controller是否在正常工作。

第5步：启动controller，执行make run。

这一步在前台运行，会占用一个终端，可以先输入几个回车，方便等会看日志。

[root@workstation guestbook]# make run
/root/projects-ok/guestbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/projects-ok/guestbook/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
1.6487131587786288e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6487131587789116e+09	INFO	setup	starting manager
1.6487131587790835e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6487131587791355e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.6487131587792726e+09	INFO	controller.guestbook	Starting EventSource	{"reconciler group": "webapp.my.domain", "reconciler kind": "Guestbook", "source": "kind source: *v1.Guestbook"}
1.6487131587792993e+09	INFO	controller.guestbook	Starting Controller	{"reconciler group": "webapp.my.domain", "reconciler kind": "Guestbook"}
1.6487131588802712e+09	INFO	controller.guestbook	Starting workers	{"reconciler group": "webapp.my.domain", "reconciler kind": "Guestbook", "worker count": 1}

第6步：打开新的终端，创建自定义资源对象，执行:kubectl apply -f config/samples/

[root@workstation guestbook]# kubectl apply -f config/samples/
guestbook.webapp.my.domain/guestbook-sample created
[root@workstation guestbook]# 

查看一下自定义资源是否成功

[root@workstation guestbook]# kubectl get guestbook
NAME               AGE
guestbook-sample   2m44s
[root@workstation guestbook]# 

第7步：查看controller，可以看到创建自定义资源对象时，打印出来的日志。

[root@workstation guestbook]# make run
/root/projects-ok/guestbook/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/root/projects-ok/guestbook/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go run ./main.go
1.6487131587786288e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": ":8080"}
1.6487131587789116e+09	INFO	setup	starting manager
1.6487131587790835e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
1.6487131587791355e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
1.6487131587792726e+09	INFO	controller.guestbook	Starting EventSource	{"reconciler group": "webapp.my.domain", "reconciler kind": "Guestbook", "source": "kind source: *v1.Guestbook"}
1.6487131587792993e+09	INFO	controller.guestbook	Starting Controller	{"reconciler group": "webapp.my.domain", "reconciler kind": "Guestbook"}
1.6487131588802712e+09	INFO	controller.guestbook	Starting workers	{"reconciler group": "webapp.my.domain", "reconciler kind": "Guestbook", "worker count": 1}





1, default/guestbook-sample
2, goroutine 183 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
my.domain/guestbook/controllers.(*GuestbookReconciler).Reconcile(0x18372f8, {0x18372f8, 0xc0005b7860}, {{{0xc00044d8a6, 0x157c260}, {0xc00044d8b0, 0x30}}})
	/root/projects-ok/guestbook/controllers/guestbook_controller.go:56 +0x125
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000376b00, {0x18372f8, 0xc0005b77a0}, {{{0xc00044d8a6, 0x157c260}, {0xc00044d8b0, 0x415594}}})
	/root/go/go-path/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114 +0x26f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000376b00, {0x1837250, 0xc000023600}, {0x14d4560, 0xc0002c4c00})
	/root/go/go-path/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311 +0x33e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000376b00, {0x1837250, 0xc000023600})
	/root/go/go-path/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/root/go/go-path/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/root/go/go-path/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:223 +0x357

第8步：修改资源，并检查contorller中的日志输出。

 

 如果controller有新的日志输出，则说明controller已经正确工作，能监听到K8S的事件。

附录：

附1：整个工程生成的文件：

[root@workstation guestbook]# tree .
.
├── api
│   └── v1
│       ├── groupversion_info.go
│       ├── guestbook_types.go
│       └── zz_generated.deepcopy.go
├── bin
│   ├── controller-gen
│   └── kustomize
├── config
│   ├── crd
│   │   ├── bases
│   │   │   └── webapp.my.domain_guestbooks.yaml
│   │   ├── kustomization.yaml
│   │   ├── kustomizeconfig.yaml
│   │   └── patches
│   │       ├── cainjection_in_guestbooks.yaml
│   │       └── webhook_in_guestbooks.yaml
│   ├── default
│   │   ├── kustomization.yaml
│   │   ├── manager_auth_proxy_patch.yaml
│   │   └── manager_config_patch.yaml
│   ├── manager
│   │   ├── controller_manager_config.yaml
│   │   ├── kustomization.yaml
│   │   └── manager.yaml
│   ├── prometheus
│   │   ├── kustomization.yaml
│   │   └── monitor.yaml
│   ├── rbac
│   │   ├── auth_proxy_client_clusterrole.yaml
│   │   ├── auth_proxy_role_binding.yaml
│   │   ├── auth_proxy_role.yaml
│   │   ├── auth_proxy_service.yaml
│   │   ├── guestbook_editor_role.yaml
│   │   ├── guestbook_viewer_role.yaml
│   │   ├── kustomization.yaml
│   │   ├── leader_election_role_binding.yaml
│   │   ├── leader_election_role.yaml
│   │   ├── role_binding.yaml
│   │   ├── role.yaml
│   │   └── service_account.yaml
│   └── samples
│       └── webapp_v1_guestbook.yaml
├── controllers
│   ├── guestbook_controller.go
│   └── suite_test.go
├── Dockerfile
├── go.mod
├── go.sum
├── hack
│   └── boilerplate.go.txt
├── main.go
├── Makefile
└── PROJECT

14 directories, 40 files
[root@workstation guestbook]# 

附2：

参考：

1)  官方教程

https://book.ku2bebuilder.io/quick-start.html

2）kubebuilder实战之二：初次体验kubebuilder

kubebuilder实战之二：初次体验kubebuilder_程序员欣宸的博客-CSDN博客_kubebuilder 实战