问题分析 之 Kubernetes的APIServer个数，对endpoints、Node、Service等 get all接口性能影响
Kubernetes在不同负载情况下，APIServer个数对不同资源 list、get等请求指令的性能影响。包括：单接口响应情况，apiserver负载情况分析对比。
现象：
apiserver 从1个变成2个，整体QPS未有提升，平均响应时间未有变化，单请求P99响应时间提升约9.1%
分析：
造成此10%，原因是发压后多并发时候，将CPU打满，导致connect Waiting时间变长；多实例缓解；
结论：
APIServer在请求QPS小于一定值（100qps），APIServer个数多少，对不同资源数量情况下，请求性能比较小。 也符合APIServer只是接口聚合、etcd 数据传递轻逻辑作用
对于线上只是对apiserver 扩容，会有约5%左右硬件叠加带来提升
* 注意点⚠️（条件）
- kubenetes版本 ： 1.18
- 硬件环境：4Core 虚拟 8vCore， 32G； 混部master节点组建
- 业务复杂性 ： 测试环境无复杂实时流量，即APIServer 无其他组合流量场景
- 受环境、网络影响，性能数据 无绝对值意义，有相对值值意义
环境准备
模拟条件
- 在kubenetes 1.18 版本下测试
- 创建类似于750个Node、10000个Pods
- 全部资源在一个namespace下（kubemark）
- 比较1个、2个APIServer下，请求流量从10qps->100qps变化
- 请求包括get all操作等操作
- 请求链路：办公网笔记本->公网->云上kubenetes master
环境构建
首先创建虚拟节点.
将kubemark k8s的config文件(任意kubemark k8s主节点的/root/.kube/config)拷贝到当前路径 执行以下命令在external k8s中创建ns及secret
kubectl create ns kubemark
kubectl create secret generic kubeconfig \
--type=Opaque --namespace=kubemark \
--from-file=kubelet.kubeconfig=config \
--from-file=kubeproxy.kubeconfig=config
创建750虚拟node
apiVersion: v1
kind: ReplicationController
metadata:
name: hollow-node
namespace: kubemark
labels:
name: hollow-node
spec:
replicas: 750
selector:
name: hollow-node
template:
metadata:
labels:
name: hollow-node
spec:
initContainers:
- name: init-inotify-limit
image: busybox:1.32
imagePullPolicy: IfNotPresent
command: ['sysctl', '-w', 'fs.inotify.max_user_instances=1000']
securityContext:
privileged: true
volumes:
- name: kubeconfig-volume
secret:
secretName: kubeconfig
- name: logs-volume
hostPath:
path: /var/log
- name: no-serviceaccount-access-to-real-master
emptyDir: {}
containers:
- name: hollow-kubelet
image: antrea/kubemark:v1.18.4
imagePullPolicy: IfNotPresent
ports:
- containerPort: 4194
- containerPort: 10250
- containerPort: 10255
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
command: [
"/kubemark",
"--morph=kubelet",
"--name=$(NODE_NAME)",
"--kubeconfig=/kubeconfig/kubelet.kubeconfig",
"--log-file=/var/log/kubelet-$(NODE_NAME).log",
"--logtostderr=false"
]
volumeMounts:
- name: kubeconfig-volume
mountPath: /kubeconfig
readOnly: true
- name: logs-volume
mountPath: /var/log
resources:
requests:
cpu: 10m
memory: 20M
securityContext:
privileged: true
- name: hollow-proxy
image: antrea/kubemark:v1.18.4
imagePullPolicy: IfNotPresent
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
command: [
"/kubemark",
"--morph=proxy",
"--name=$(NODE_NAME)",
"--kubeconfig=/kubeconfig/kubeproxy.kubeconfig",
"--log-file=/var/log/kubeproxy-$(NODE_NAME).log",
"--logtostderr=false"
]
volumeMounts:
- name: kubeconfig-volume
mountPath: /kubeconfig
readOnly: true
- name: logs-volume
mountPath: /var/log
resources:
requests:
cpu: 10m
memory: 20M
结果
......
hollow-node-zq2r5 Ready <none> 26m v1.18.4-dirty
hollow-node-zrbxj Ready <none> 8m13s v1.18.4-dirty
hollow-node-zrcmk Ready <none> 3m13s v1.18.4-dirty
hollow-node-zrfhg Ready <none> 25m v1.18.4-dirty
hollow-node-zsx6d Ready <none> 25m v1.18.4-dirty
hollow-node-ztzmp Ready <none> 7m3s v1.18.4-dirty
hollow-node-zvwzk Ready <none> 4m23s v1.18.4-dirty
hollow-node-zxz2d Ready <none> 14m v1.18.4-dirty
hollow-node-zzh9t Ready <none> 25m v1.18.4-dirty
hollow-node-zzr7s Ready <none> 8m41s v1.18.4-dirty
kcs-mysql-k8s-m-xxxx Ready master 179m v1.18.3
kcs-mysql-k8s-m-shpfx Ready master 3h1m v1.18.3
kcs-mysql-k8s-m-xj7rh Ready master 177m v1.18.3
kcs-mysql-k8s-s-8jb49 Ready <none> 174m v1.18.3
kcs-mysql-k8s-s-b75sd Ready <none> 174m v1.18.3
kcs-mysql-k8s-s-gsd5d Ready <none> 175m v1.18.3
kcs-mysql-k8s-s-gtvr7 Ready <none> 175m v1.18.3
kcs-mysql-k8s-s-h8lcx Ready <none> 174m v1.18.3
kcs-mysql-k8s-s-k2hlm Ready <none> 174m v1.18.3
kcs-mysql-k8s-s-m5csq Ready <none> 174m v1.18.3
kcs-mysql-k8s-s-qfd6m Ready <none> 174m v1.18.3
kcs-mysql-k8s-s-rwbfp Ready <none> 174m v1.18.3
kcs-mysql-k8s-s-sqk2k Ready <none> 174m v1.18.3
[root@kcs-mysql-k8s-m-xxxx ~]# kubectl get node | wc -l
764
添加 10000个虚拟Pods 并且亲和到虚拟node上
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-deployment
namespace: kubemark
labels:
app: myapp
version: v1
spec:
replicas: 10000
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: app
image: docker.io/busybox:latest
imagePullPolicy: IfNotPresent
command: ['sleep', '3600']
securityContext:
privileged: true
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution: # 硬策略
nodeSelectorTerms:
- matchExpressions:
- key: node-role.kubernetes.io/node
operator: NotIn
values:
- "true"
结果：
myapp-deployment-8467886dbd-zzfbd 1/1 Running 0 90m
myapp-deployment-8467886dbd-zzhlg 1/1 Running 0 100m
myapp-deployment-8467886dbd-zzkvq 1/1 Running 0 90m
myapp-deployment-8467886dbd-zzmch 1/1 Running 0 89m
myapp-deployment-8467886dbd-zzmdh 1/1 Running 0 96m
myapp-deployment-8467886dbd-zzmhn 1/1 Running 0 99m
myapp-deployment-8467886dbd-zzp8r 1/1 Running 0 100m
myapp-deployment-8467886dbd-zzqhf 1/1 Running 0 87m
myapp-deployment-8467886dbd-zzrws 1/1 Running 0 87m
myapp-deployment-8467886dbd-zztcm 1/1 Running 1 94m
myapp-deployment-8467886dbd-zzv4r 1/1 Running 1 94m
myapp-deployment-8467886dbd-zzvt8 1/1 Running 0 95m
myapp-deployment-8467886dbd-zzvwx 1/1 Running 0 86m
[root@kcs-mysql-k8s-m-xxxx ~]# kubectl get pods --namespace=kubemark | grep "deployment" | wc -l
10000
模拟压测 http请求
单APIServer节点压测
使用 proxy 代理
[root@kcs-mysql-k8s-m-xxxx ~]# kubectl proxy --port=8080
Starting to serve on 127.0.0.1:8080
使用一个APIServer请求 1000Node 和 10000Pod情况
- 1000Nodes Get ALL全量请求（CPU使用率80%）
[root@kcs-mysql-k8s-m-xxxx /]# ab -n 500 -c 20 http://localhost:8080/api/v1/nodes
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
Server Software:
Server Hostname: localhost
Server Port: 8080
Document Path: /api/v1/nodes
Document Length: 3336918 bytes
Concurrency Level: 20
Time taken for tests: 10.656 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Total transferred: 1668531500 bytes
HTML transferred: 1668459000 bytes
Requests per second: 46.92 [#/sec] (mean)
Time per request: 426.247 [ms] (mean)
Time per request: 21.312 [ms] (mean, across all concurrent requests)
Transfer rate: 152909.06 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.9 0 14
Processing: 100 418 193.9 380 1147
Waiting: 73 361 196.2 317 1110
Total: 100 418 193.9 380 1147
Percentage of the requests served within a certain time (ms)
50% 387
66% 482
75% 556
80% 605
90% 739
95% 840
98% 949
99% 1029
100% 1501 (longest request)
- 10000个Pod Get ALL 全量请求（CPU使用率99%）
[root@kcs-mysql-k8s-m-xxxx /]# ab -n 100 -c 5 http://localhost:8080/api/v1/namespaces/kubemark/pods
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient).....done
Server Software:
Server Hostname: localhost
Server Port: 8080
Document Path: /api/v1/namespaces/kubemark/pods
Document Length: 48362569 bytes
Concurrency Level: 5
Time taken for tests: 33.654 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 4836271400 bytes
HTML transferred: 4836256900 bytes
Requests per second: 2.97 [#/sec] (mean)
Time per request: 1682.684 [ms] (mean)
Time per request: 336.537 [ms] (mean, across all concurrent requests)
Transfer rate: 140338.90 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1198 1638 217.8 1599 2254
Waiting: 777 1192 214.3 1159 1825
Total: 1198 1638 217.8 1599 2255
Percentage of the requests served within a certain time (ms)
50% 1599
66% 1716
75% 1764
80% 1875
90% 1926
95% 2054
98% 2186
99% 2255
100% 2255 (longest request)
双APIServer实例负载
通过Nginx代理双APIServer;
upstream proxybb {
server 192.168.10.31:13331;
server 192.168.10.32:13331;
}
server {
listen 13332;
access_log /var/log/nginx/host.access.log main;
error_log /var/log/nginx/host.error.log;
location ~ / {
proxy_pass http://proxybb;
}
- 1000Nodes Get ALL全量请求（CPU使用率50%）
[root@kcs-mysql-k8s-m-xxxx /]# ab -n 500 -c 20 http://localhost:13332/api/v1/nodes
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests
Server Software:
Server Hostname: localhost
Server Port: 13332
Document Path: /api/v1/nodes
Document Length: 3336918 bytes
Concurrency Level: 20
Time taken for tests: 11.026 seconds
Complete requests: 500
Failed requests: 0
Write errors: 0
Total transferred: 1668531500 bytes
HTML transferred: 1668459000 bytes
Requests per second: 45.35 [#/sec] (mean)
Time per request: 441.030 [ms] (mean)
Time per request: 22.051 [ms] (mean, across all concurrent requests)
Transfer rate: 147783.68 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.1 0 1
Processing: 91 433 215.4 387 1501
Waiting: 69 374 204.7 330 1425
Total: 91 433 215.4 387 1501
Percentage of the requests served within a certain time (ms)
50% 380
66% 485
75% 553
80% 589
90% 702
95% 779
98% 853
99% 928
100% 1147 (longest request)
- 10000个Pod Get ALL 全量请求（CPU使用率80%）
[root@kcs-mysql-k8s-m-xxxx /]# ab -n 100 -c 5 http://localhost:13332/api/v1/namespaces/kubemark/pods
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient).....done
Server Software:
Server Hostname: localhost
Server Port: 13332
Document Path: /api/v1/namespaces/kubemark/pods
Document Length: 48362569 bytes
Concurrency Level: 5
Time taken for tests: 33.189 seconds
Complete requests: 100
Failed requests: 0
Write errors: 0
Total transferred: 4836271400 bytes
HTML transferred: 4836256900 bytes
Requests per second: 3.01 [#/sec] (mean)
Time per request: 1659.453 [ms] (mean)
Time per request: 331.891 [ms] (mean, across all concurrent requests)
Transfer rate: 142303.52 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 1107 1604 187.7 1598 2040
Waiting: 752 1164 188.8 1184 1568
Total: 1107 1604 187.7 1598 2040
Percentage of the requests served within a certain time (ms)
50% 1598
66% 1703
75% 1738
80% 1765
90% 1864
95% 1944
98% 1980
99% 2040
100% 2040 (longest request)
「如果这篇文章对你有用,请随意打赏」
如果这篇文章对你有用,请随意打赏
使用微信扫描二维码完成支付
