一、测试目标
1、集群容量上限：Node总数和Pod总数两个维度
2、性能瓶颈
3、服务延迟

k8s集群性能测试-运用kubemark

[root@10-2-19-17 ~]# cd $GOPATH/src/k8s.io/kubernetes

[root@10-2-19-17 kubernetes]# ./hack/generate-bindata.sh
~/gocode/src/k8s.io/kubernetes ~/gocode/src/k8s.io/kubernetes
Generated bindata file : test/e2e/generated/bindata.go has 13126 test/e2e/generated/bindata.go lines of lovely automated artifacts
No changes in generated bindata file: pkg/generated/bindata.go
~/gocode/src/k8s.io/kubernetes

[root@10-2-19-17 kubernetes]# make WHAT="test/e2e/e2e.test"
+++ [1127 17:07:39] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1127 17:07:39] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/gocode/src/k8s.io/kubernetes ~/gocode/src/k8s.io/kubernetes/test/e2e/generated
~/gocode/src/k8s.io/kubernetes/test/e2e/generated
+++ [1127 17:07:40] Building go targets for linux/amd64:
    test/e2e/e2e.test

[root@10-2-19-17 kubernetes]# make ginkgo
+++ [1127 17:09:31] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [1127 17:09:31] Generating bindata:
    test/e2e/generated/gobindata_util.go
~/gocode/src/k8s.io/kubernetes ~/gocode/src/k8s.io/kubernetes/test/e2e/generated
~/gocode/src/k8s.io/kubernetes/test/e2e/generated
+++ [1127 17:09:32] Building go targets for linux/amd64:
    vendor/github.com/onsi/ginkgo/ginkgo

[root@10-2-19-17 kubernetes]# export KUBECTL_PATH=/usr/bin/kubectl

创建kubeconfig

cp /etc/kubernetes/admin.conf  /root/.kube/
mv admin.conf config

执行performance test

[root@10-2-20-7 kubernetes]#  go run hack/e2e.go -v --test  --test_args="--host=https://10.2.20.5:6443 --ginkgo.focus=\[Feature:Performance\]" >> logs/log.txt 
flag provided but not defined: -v
Usage of /tmp/go-build327486475/command-line-arguments/_obj/exe/e2e:
  -get
    	go get -u kubetest if old or not installed (default true)
  -old duration
    	Consider kubetest old if it exceeds this (default 24h0m0s)
2017/11/30 10:13:10 e2e.go:54: NOTICE: go run hack/e2e.go is now a shim for test-infra/kubetest
2017/11/30 10:13:10 e2e.go:55:   Usage: go run hack/e2e.go [--get=true] [--old=24h0m0s] -- [KUBETEST_ARGS]
2017/11/30 10:13:10 e2e.go:56:   The separator is required to use --get or --old flags
2017/11/30 10:13:10 e2e.go:57:   The -- flag separator also suppresses this message
2017/11/30 10:13:10 e2e.go:76: Calling kubetest -v --test --test_args=--host=https://10.2.20.5:6443 --ginkgo.focus=\[Feature:Performance\]...
2017/11/30 10:13:10 util.go:422: Please use kubetest --provider=local (instead of deprecated KUBERNETES_PROVIDER=local)
2017/11/30 10:13:10 util.go:155: Running: ./cluster/kubectl.sh --match-server-version=false version
2017/11/30 10:13:10 util.go:157: Step './cluster/kubectl.sh --match-server-version=false version' finished in 235.717856ms
2017/11/30 10:13:10 util.go:155: Running: ./hack/e2e-internal/e2e-status.sh
Local doesn't need special preparations for e2e tests
2017/11/30 10:13:10 util.go:157: Step './hack/e2e-internal/e2e-status.sh' finished in 279.000773ms
2017/11/30 10:13:10 util.go:155: Running: ./hack/ginkgo-e2e.sh --host=https://10.2.20.5:6443 --ginkgo.focus=\[Feature:Performance\]
Local doesn't need special preparations for e2e tests
2017/11/30 10:18:34 util.go:157: Step './hack/ginkgo-e2e.sh --host=https://10.2.20.5:6443 --ginkgo.focus=\[Feature:Performance\]' finished in 5m23.832173249s
2017/11/30 10:18:34 e2e.go:80: Done
[root@10-2-20-7 kubernetes]# 

输出日志：

...
[Finish:Performance]
[AfterEach] [k8s.io] Load capacity
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:120
Nov 30 10:18:04.052: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "e2e-tests-load-phvcx" for this suite.
Nov 30 10:18:14.161: INFO: namespace: e2e-tests-load-phvcx, resource: bindings, ignored listing per whitelist
STEP: Destroying namespace "e2e-tests-load-30-nodepods-1-4h4pc" for this suite.
Nov 30 10:18:34.339: INFO: namespace: e2e-tests-load-30-nodepods-1-4h4pc, resource: bindings, ignored listing per whitelist

??[SLOW TEST:181.733 seconds]
[k8s.io] Load capacity
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:650
  [Feature:Performance] should be able to handle 30 pods per node { ReplicationController} with 0 secrets and 0 daemons
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/load.go:265
------------------------------
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSNov 30 10:18:34.455: INFO: Running AfterSuite actions on all node
Nov 30 10:18:34.455: INFO: Running AfterSuite actions on node 1

Ran 2 of 591 Specs in 323.273 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 589 Skipped PASS

Ginkgo ran 1 suite in 5m23.72874212s
Test Suite Passed

测试过程中会产生两个namespace: 完成后会删除

[root@10-2-20-7 e2e]# kubectl get namespace
NAME                           STATUS        AGE
default                        Active        36d
e2e-tests-density-30-1-vmjnw   Active        1m
e2e-tests-density-cbzmb        Active        1m

默认情况下，在每个测试结束时，它将删除Kubemark Master上的命名空间和它下面的所有东西（例如事件，复制控制器），这会花费很多时间。大多数情况下不需要这样的工作：如果您在运行后删除了Kubemark集群run-e2e-tests.sh，你不关心命名空间的删除性能，具体涉及到etcd; 有一个标志，使您可以避免名称空间删除：--delete-namespace=false。添加标志应该让你看到日志：Found DeleteNamespace=false, skipping namespace deletion!

中途出现错误按ctrl+c 结束，用下面命令清理：

go run hack/e2e.go -v --down

注意：如果您在本地群集上重复运行测试，则可能需要定期执行一些手动清理：

rm -rf /var/run/kubernetes #清除kube生成的凭据，有时陈旧的权限可能会导致问题。
sudo iptables -F #清除kube-proxy所留下的IP表规则。

参考：

 

https://supereagle.github.io/2017/03/09/kubemark/ 
https://www.tuicool.com/articles/BBbIBzq 

https://www.cnblogs.com/opama/p/5931515.html