【Helm】release太多导致报错,一次从问题排查,修改源码编译到构建tiller镜像修复的经历...

 

0. 起因

在我们测试环境k8s上运行了数十个helm release, 昨天helm ls的时候居然报错了

 

  1. $ helm ls

  2. Error: trying to send message larger than max (23934705 vs. 20971520)

这个问题搜索下, 发现github issue今年一月就已经有人遇到了 https://github.com/helm/helm/issues/3322 , 今年二月份官方的同学也提pr修复了 https://github.com/helm/helm/pull/3514

为啥我这里还会出现呢

1. 探究

image
image
看下修复pr的主要内容, 发现只是把限制从4M提升到20M...

这个值还是一个常量, 目前最近的release和master也一样 我们使用的tiller镜像 registry.cn-beijing.aliyuncs.com/acs/tiller:v2.9.1，已经包了修复内容,看来想解决只能自己修改源码编译了

2. 修复

2.1 修改源码

image

const maxMsgSize = 1024 * 1024 * 100
这里改成100M,还能超的话只能说要再来一个集群了..

2.2 编译 Helm

拉下代码, 发现helm使用make进行构建, 按照官方 https://docs.helm.sh/developers/, 尝试初始化和构建

 

  1. 17:03 $ make bootstrap build

  2. glide install --strip-vendor

  3. [INFO] Downloading dependencies. Please wait...

  4. [INFO] --> Found desired version locally github.com/aokoli/goutils 9c37978a95bd5c709a15883b6242714ea6709e64!

  5. [INFO] --> Found desired version locally github.com/asaskevich/govalidator 7664702784775e51966f0885f5cd27435916517b!

  6. [INFO] --> Found desired version locally github.com/Azure/go-ansiterm d6e3b3328b783f23731bc4d058875b0371ff8109!

  7. [INFO] --> Found desired version locally github.com/Azure/go-autorest bca49d5b51a50dc5bb17bbf6204c711c6dbded06!

  8. [INFO] --> Found desired version locally github.com/beorn7/perks 3ac7bf7a47d159a033b107610db8a1b6575507a4!

  9. [INFO] --> Found desired version locally github.com/BurntSushi/toml 3012a1dbe2e4bd1391d42b32f0577cb7bbc7f005!

  10. [INFO] --> Found desired version locally github.com/chai2010/gettext-go c6fed771bfd517099caf0f7a961671fa8ed08723!

  11. [INFO] --> Found desired version locally github.com/cpuguy83/go-md2man 71acacd42f85e5e82f70a55327789582a5200a90!

  12. [INFO] --> Found desired version locally github.com/cyphar/filepath-securejoin a261ee33d7a517f054effbf451841abaafe3e0fd!

  13. [INFO] --> Found desired version locally github.com/davecgh/go-spew 782f4967f2dc4564575ca782fe2d04090b5faca8!

  14. [INFO] --> Found desired version locally github.com/dgrijalva/jwt-go 01aeca54ebda6e0fbfafd0a524d234159c05ec20!

  15. [INFO] --> Found desired version locally github.com/docker/distribution edc3ab29cdff8694dd6feb85cfeb4b5f1b38ed9c!

  16. [INFO] --> Found desired version locally github.com/docker/docker a9fbbdc8dd8794b20af358382ab780559bca589d!

  17. [INFO] --> Found desired version locally github.com/docker/go-connections 3ede32e2033de7505e6500d6c868c2b9ed9f169d!

  18. [INFO] --> Found desired version locally github.com/docker/go-units 9e638d38cf6977a37a8ea0078f3ee75a7cdb2dd1!

  19. [INFO] --> Found desired version locally github.com/docker/spdystream 449fdfce4d962303d702fec724ef0ad181c92528!

  20. [INFO] --> Found desired version locally github.com/evanphx/json-patch 36442dbdb585210f8d5a1b45e67aa323c197d5c4!

  21. [INFO] --> Found desired version locally github.com/exponent-io/jsonpath d6023ce2651d8eafb5c75bb0c7167536102ec9f5!

  22. [INFO] --> Found desired version locally github.com/fatih/camelcase f6a740d52f961c60348ebb109adde9f4635d7540!

  23. [INFO] --> Found desired version locally github.com/ghodss/yaml 73d445a93680fa1a78ae23a5839bad48f32ba1ee!

  24. [INFO] --> Found desired version locally github.com/go-openapi/jsonpointer 46af16f9f7b149af66e5d1bd010e3574dc06de98!

  25. [INFO] --> Found desired version locally github.com/go-openapi/jsonreference 13c6e3589ad90f49bd3e3bbe2c2cb3d7a4142272!

  26. [INFO] --> Found desired version locally github.com/go-openapi/spec 1de3e0542de65ad8d75452a595886fdd0befb363!

  27. [INFO] --> Found desired version locally github.com/go-openapi/swag f3f9494671f93fcff853e3c6e9e948b3eb71e590!

  28. [INFO] --> Found desired version locally github.com/gobwas/glob 5ccd90ef52e1e632236f7326478d4faa74f99438!

  29. [INFO] --> Found desired version locally github.com/gogo/protobuf c0656edd0d9eab7c66d1eb0c568f9039345796f7!

  30. [INFO] --> Found desired version locally github.com/golang/glog 44145f04b68cf362d9c4df2182967c2275eaefed!

  31. [INFO] --> Found desired version locally github.com/golang/groupcache 02826c3e79038b59d737d3b1c0a1d937f71a4433!

  32. [INFO] --> Found desired version locally github.com/golang/protobuf 1643683e1b54a9e88ad26d98f81400c8c9d9f4f9!

  33. [INFO] --> Found desired version locally github.com/google/btree 7d79101e329e5a3adf994758c578dab82b90c017!

  34. [INFO] --> Found desired version locally github.com/google/gofuzz 44d81051d367757e1c7c6a5a86423ece9afcf63c!

  35. [INFO] --> Found desired version locally github.com/google/uuid 064e2069ce9c359c118179501254f67d7d37ba24!

  36. [INFO] --> Found desired version locally github.com/googleapis/gnostic 0c5108395e2debce0d731cf0287ddf7242066aba!

  37. [INFO] --> Found desired version locally github.com/gophercloud/gophercloud 781450b3c4fcb4f5182bcc5133adb4b2e4a09d1d!

  38. [INFO] --> Found desired version locally github.com/gosuri/uitable 36ee7e946282a3fb1cfecd476ddc9b35d8847e42!

  39. [INFO] --> Found desired version locally github.com/gregjones/httpcache 787624de3eb7bd915c329cba748687a3b22666a6!

  40. [INFO] --> Found desired version locally github.com/grpc-ecosystem/go-grpc-prometheus 0c1b191dbfe51efdabe3c14b9f6f3b96429e0722!

  41. [INFO] --> Found desired version locally github.com/hashicorp/golang-lru a0d98a5f288019575c6d1f4bb1573fef2d1fcdc4!

  42. [INFO] --> Found desired version locally github.com/huandu/xstrings 3959339b333561bf62a38b424fd41517c2c90f40!

  43. [INFO] --> Found desired version locally github.com/imdario/mergo 9316a62528ac99aaecb4e47eadd6dc8aa6533d58!

  44. [INFO] --> Found desired version locally github.com/inconshreveable/mousetrap 76626ae9c91c4f2a10f34cad8ce83ea42c93bb75!

  45. [INFO] --> Found desired version locally github.com/json-iterator/go f2b4162afba35581b6d4a50d3b8f34e33c144682!

  46. [INFO] --> Found desired version locally github.com/mailru/easyjson 2f5df55504ebc322e4d52d34df6a1f5b503bf26d!

  47. [INFO] --> Found desired version locally github.com/MakeNowJust/heredoc bb23615498cded5e105af4ce27de75b089cbe851!

  48. [INFO] --> Found desired version locally github.com/Masterminds/semver 517734cc7d6470c0d07130e40fd40bdeb9bcd3fd!

  49. [INFO] --> Found desired version locally github.com/Masterminds/sprig 15f9564e7e9cf0da02a48e0d25f12a7b83559aa6!

  50. [WARN] Unable to checkout golang.org/x/text

  51. [ERROR] Update failed for golang.org/x/text: Cannot detect VCS

  52. [WARN] Unable to checkout golang.org/x/time

  53. [ERROR] Update failed for golang.org/x/time: Cannot detect VCS

  54. [WARN] Unable to checkout google.golang.org/appengine

  55. [ERROR] Update failed for google.golang.org/appengine: Cannot detect VCS

  56. [WARN] Unable to checkout google.golang.org/genproto

  57. [ERROR] Update failed for google.golang.org/genproto: Cannot detect VCS

  58. [WARN] Unable to checkout google.golang.org/grpc

  59. [ERROR] Update failed for google.golang.org/grpc: Cannot detect VCS

  60. [ERROR] Failed to install: Cannot detect VCS

  61. Cannot detect VCS

  62. Cannot detect VCS

  63. Cannot detect VCS

  64. Cannot detect VCS

  65. Cannot detect VCS

  66. Cannot detect VCS

  67. Cannot detect VCS

  68. Cannot detect VCS

  69. Cannot detect VCS

  70. make: *** [bootstrap] Error 1

问题1: 国内golang的依赖包各种被墙

 

  1. .PHONY: bootstrap

  2. bootstrap:

  3. ifndef HAS_GLIDE

  4. go get -u github.com/Masterminds/glide

  5. endif

  6. ifndef HAS_GOX

  7. go get -u github.com/mitchellh/gox

  8. endif

  9.  

  10. ifndef HAS_GIT

  11. $(error You must install Git)

  12. endif

  13. glide install --strip-vendor

  14. go build -o bin/protoc-gen-go ./vendor/github.com/golang/protobuf/protoc-gen-go

看一眼Makefile, 这里用的glide管理的依赖包

好在github基本都会有对应的包源码, 有几种解决方式:

  1. clone源码然后按照目录结构放到对应$GOPATH/src下
  2. 给git设置proxy,小飞机你懂的~
  3. 使用 gopm
  4. 使用glide的mirror功能设置映射 glide-mirror

这里用的方法4, mirrors.yaml, 不想一个一个做映射可以参考这里 mirrors.yaml

问题2: 编译路径

 

  1. 17:36 $ make build

  2. GOBIN=/home/jokimina/golang/k8s.io/helm/bin go install -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.GitCommit=26c88ec815785708cc19c014366ce4bb000ae89c -X k8s.io/helm/pkg/version.GitTreeState=dirty' k8s.io/helm/cmd/...

  3. warning: "k8s.io/helm/cmd/..." matched no packages

如果是直接拉下代码放到随意的目录, build的时候会报错, 需要放到
$GOPATH/src/k8s.io/helm再执行编译

问题3: golang默认不是静态编译, docker alpine镜像没有运行所需要的库

 

  1. $ make build

  2. GOBIN=/home/xxx/go/src/k8s.io/helm/bin CGO_ENABLED=0 go install -tags '' -ldflags '-w -s -X k8s.io/helm/pkg/version.GitCommit=147c8217c7843b53e98528f6df12890d64c696be -X k8s.io/helm/pkg/version.GitTreeState=dirty' k8s.io/helm/cmd/...

  3. ~/go/src/k8s.io/helm

  4. $ ls bin/

  5. helm rudder tiller

make bootstrap build执行完后会生成二进制文件到 $GOBIN 目录下 , 直接替换掉阿里云tiller容器内的运行发现报错

 

  1. # /tiller

  2. sh: /tiller: not found

但是本地可以运行, 搜索下 https://stackoverflow.com/questions/36279253/go-compiled-binary-wont-run-in-an-alpine-docker-container-on-ubuntu-host

发现是由于默认没有静态编译导致引用了alpine不存在的库链接

 

  1. # ldd tiller

  2. /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)

  3. libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)

  4. libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x55d98fa73000)

重新静态编译下, 看到Makefile默认支持交叉编译这里偷个懒

image

make TARGETS="linux/amd64" build-cross

节省时间这里就只编译需要的了, 构建好的二进制文件会在 _dist/下

Logo
K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 |  韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号​前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。​长夜有穷，真水无香。领略过了 K8s“身在江

avatar K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub

avatar K8S/Kubernetes

k8s 火了！

2020，上云之年，产品云端化成为一种趋势。在一线城市，很多公司都已经构建了自己的私有云环境，比如阿里云、网易云、华为云等。而Kubernetes 作为基于容器编排领域的王者，具备扩展...

avatar K8S/Kubernetes