• Kubernetes测试
    • 单元测试
    • Benchmark测试
    • 集成测试
    • End to end (e2e)测试
    • Fedaration e2e测试
    • Node e2e测试
    • 补充说明
    • kubernetes测试工具集test-infra
    • 参考文档

    Kubernetes测试

    单元测试

    单元测试仅依赖于源代码,是测试代码逻辑是否符合预期的最简单方法。

    运行所有的单元测试

    1. make test

    仅测试指定的package

    1. # 单个package
    2. make test WHAT=./pkg/api
    3. # 多个packages
    4. make test WHAT=./pkg/{api,kubelet}

    或者,也可以直接用go test

    1. go test -v k8s.io/kubernetes/pkg/kubelet

    仅测试指定package的某个测试case

    1. # Runs TestValidatePod in pkg/api/validation with the verbose flag set
    2. make test WHAT=./pkg/api/validation KUBE_GOFLAGS="-v" KUBE_TEST_ARGS='-run ^TestValidatePod$'
    3. # Runs tests that match the regex ValidatePod|ValidateConfigMap in pkg/api/validation
    4. make test WHAT=./pkg/api/validation KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ValidatePod\|ValidateConfigMap$"

    或者直接用go test

    1. go test -v k8s.io/kubernetes/pkg/api/validation -run ^TestValidatePod$

    并行测试

    并行测试是root out flakes的一种有效方法:

    1. # Have 2 workers run all tests 5 times each (10 total iterations).
    2. make test PARALLEL=2 ITERATION=5

    生成测试报告

    1. make test KUBE_COVER=y

    Benchmark测试

    1. go test ./pkg/apiserver -benchmem -run=XXX -bench=BenchmarkWatch

    集成测试

    Kubernetes集成测试需要安装etcd(只要按照即可,不需要启动),比如

    1. hack/install-etcd.sh # Installs in ./third_party/etcd
    2. echo export PATH="\$PATH:$(pwd)/third_party/etcd" >> ~/.profile # Add to PATH

    集成测试会在需要的时候自动启动etcd和kubernetes服务,并运行test/integration里面的测试。

    运行所有集成测试

    1. make test-integration # Run all integration tests.

    指定集成测试用例

    1. # Run integration test TestPodUpdateActiveDeadlineSeconds with the verbose flag set.
    2. make test-integration KUBE_GOFLAGS="-v" KUBE_TEST_ARGS="-run ^TestPodUpdateActiveDeadlineSeconds$"

    End to end (e2e)测试

    End to end (e2e) 测试模拟用户行为操作Kubernetes,用来保证Kubernetes服务或集群的行为完全符合设计预期。

    在开启e2e测试之前,需要先编译测试文件,并设置KUBERNETES_PROVIDER(默认为gce):

    1. make WHAT='test/e2e/e2e.test'
    2. make ginkgo
    3. export KUBERNETES_PROVIDER=local

    启动cluster,测试,最后停止cluster

    1. # build Kubernetes, up a cluster, run tests, and tear everything down
    2. go run hack/e2e.go -- -v --build --up --test --down

    仅测试指定的用例

    1. go run hack/e2e.go -v -test --test_args='--ginkgo.focus=Kubectl\sclient\s\[k8s\.io\]\sKubectl\srolling\-update\sshould\ssupport\srolling\-update\sto\ssame\simage\s\[Conformance\]$'

    略过测试用例

    1. go run hack/e2e.go -- -v --test --test_args="--ginkgo.skip=Pods.*env

    并行测试

    1. # Run tests in parallel, skip any that must be run serially
    2. GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.skip=\[Serial\]"
    3. # Run tests in parallel, skip any that must be run serially and keep the test namespace if test failed
    4. GINKGO_PARALLEL=y go run hack/e2e.go --v --test --test_args="--ginkgo.skip=\[Serial\] --delete-namespace-on-failure=false"

    清理测试

    1. go run hack/e2e.go -- -v --down

    有用的-ctl

    1. # -ctl can be used to quickly call kubectl against your e2e cluster. Useful for
    2. # cleaning up after a failed test or viewing logs. Use -v to avoid suppressing
    3. # kubectl output.
    4. go run hack/e2e.go -- -v -ctl='get events'
    5. go run hack/e2e.go -- -v -ctl='delete pod foobar'

    Fedaration e2e测试

    1. export FEDERATION=true
    2. export E2E_ZONES="us-central1-a us-central1-b us-central1-f"
    3. # or export FEDERATION_PUSH_REPO_BASE="quay.io/colin_hom"
    4. export FEDERATION_PUSH_REPO_BASE="gcr.io/${GCE_PROJECT_NAME}"
    5. # build container images
    6. KUBE_RELEASE_RUN_TESTS=n KUBE_FASTBUILD=true go run hack/e2e.go -- -v -build
    7. # push the federation container images
    8. build/push-federation-images.sh
    9. # Deploy federation control plane
    10. go run hack/e2e.go -- -v --up
    11. # Finally, run the tests
    12. go run hack/e2e.go -- -v --test --test_args="--ginkgo.focus=\[Feature:Federation\]"
    13. # Don't forget to teardown everything down
    14. go run hack/e2e.go -- -v --down

    可以用cluster/log-dump.sh <directory>方便的下载相关日志,帮助排查测试中碰到的问题。

    Node e2e测试

    Node e2e仅测试Kubelet的相关功能,可以在本地或者集群中测试

    1. export KUBERNETES_PROVIDER=local
    2. make test-e2e-node FOCUS="InitContainer"
    3. make test_e2e_node TEST_ARGS="--experimental-cgroups-per-qos=true"

    补充说明

    借助kubectl的模版可以方便获取想要的数据,比如查询某个container的镜像的方法为

    1. kubectl get pods nginx-4263166205-ggst4 -o template '--template={{if (exists . "status" "containerStatuses")}}{{range .status.containerStatuses}}{{if eq .name "nginx"}}{{.image}}{{end}}{{end}}{{end}}'

    kubernetes测试工具集test-infra

    test-infra是由kubernetes官方开源的测试框架,其中包括了Kubernetes测试工具集和测试结果展示。下图展示了test-infra的架构:

    test-infra架构图(图片来自官方GitHub)

    该测试框架主要是真多Google公有云做的,支持kubernetes1.6以上版本的测试。详见https://github.com/kubernetes/test-infra。

    参考文档

    • https://github.com/kubernetes/test-infra