Kubernetes 自定义指标hpa实现：prometheus-adapter
如何实现自定义指标采集
其中包括：
- 自定义指标暴露，并采集
- 完成将原生自定义指标转换为 apiserver aggregation 指标
- 实现hpa指标名称转换生成
实现部署
前提
部署完成Prometheus, 可通过 Prometheus 获得可用tag
部署adapter
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install my-release prometheus-community/prometheus-adapter
检查部署情况：
dongjiang@MacBook Pro:~ $ kubectl get pod -n monitoring | grep "adapter"
metrics-prometheus-adapter-6ccb5fb984-qr76k 1/1 Running 6 3d21h
APIService检查
dongjiang@MacBook Pro:~ $ kubectl get apiservice
NAME SERVICE AVAILABLE AGE
v1. Local True 39d
v1.admissionregistration.k8s.io Local True 39d
v1.apiextensions.k8s.io Local True 39d
v1.apps Local True 39d
v1.authentication.k8s.io Local True 39d
v1.authorization.k8s.io Local True 39d
v1.autoscaling Local True 39d
v1.batch Local True 39d
v1.certificates.k8s.io Local True 39d
v1.coordination.k8s.io Local True 39d
v1.discovery.k8s.io Local True 39d
v1.events.k8s.io Local True 39d
v1.monitoring.coreos.com Local True 4d
v1.networking.k8s.io Local True 39d
v1.node.k8s.io Local True 39d
v1.policy Local True 39d
v1.rbac.authorization.k8s.io Local True 39d
v1.scheduling.k8s.io Local True 39d
v1.storage.k8s.io Local True 39d
v1alpha1.monitoring.coreos.com Local True 4d
v1beta1.custom.metrics.k8s.io monitoring/metrics-prometheus-adapter True 4d
v1beta2.flowcontrol.apiserver.k8s.io Local True 39d
v1beta3.flowcontrol.apiserver.k8s.io Local True 39d
v2.autoscaling Local True 39d
dongjiang@MacBook Pro:~ $ kubectl get apiservice v1beta1.custom.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apiregistration.k8s.io/v1","kind":"APIService","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"metrics","app.kubernetes.io/instance":"metrics","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"prometheus-adapter","app.kubernetes.io/part-of":"prometheus-adapter","app.kubernetes.io/version":"v0.11.1","helm.sh/chart":"prometheus-adapter-4.7.1"},"name":"v1beta1.custom.metrics.k8s.io"},"spec":{"group":"custom.metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"metrics-prometheus-adapter","namespace":"monitoring"},"version":"v1beta1","versionPriority":100}}
creationTimestamp: "2024-10-24T06:20:19Z"
labels:
app.kubernetes.io/component: metrics
app.kubernetes.io/instance: metrics
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: prometheus-adapter
app.kubernetes.io/part-of: prometheus-adapter
app.kubernetes.io/version: v0.11.1
helm.sh/chart: prometheus-adapter-4.7.1
name: v1beta1.custom.metrics.k8s.io
resourceVersion: "687968"
uid: 858ac5a5-86cc-47f6-b629-ce7440609dce
spec:
group: custom.metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-prometheus-adapter
namespace: monitoring
port: 443
version: v1beta1
versionPriority: 100
status:
conditions:
- lastTransitionTime: "2024-10-24T15:58:13Z"
message: all checks passed
reason: Passed
status: "True"
type: Available
部署demo
下载 echo demo helm cha service
部署 echo demo， 其中包括servicemonitor采集资源：
dongjiang@MacBook Pro:echo $ helm upgrade -f values.yaml echo .
Release "echo" has been upgraded. Happy Helming!
NAME: echo
LAST DEPLOYED: Mon Oct 28 17:06:43 2024
NAMESPACE: default
STATUS: deployed
REVISION: 4
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=echo,app.kubernetes.io/instance=echo" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
查看:
dongjiang@MacBook Pro:helm $ kubectl get all -n default | grep echo
pod/echo-7896cd67b6-gnm7k 1/1 Running 0 22m
service/echo ClusterIP 10.96.59.94 <none> 9445/TCP 22m
deployment.apps/echo 1/1 1 1 22m
replicaset.apps/echo-7896cd67b6 1 1 1 22m
horizontalpodautoscaler.autoscaling/echo Deployment/echo 1%/80% 1 100 1 22m
检查Prometheus采集指标
application_server_* 指标都是业务自定义指标
配置 promehteus adapter
查询目前是否有关于application指标，结果没有
dongjiang@MacBook Pro:echo $ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . ｜grep "application"
jq: error: Could not open file ｜grep: No such file or directory
jq: error: Could not open file application: No such file or directory
添加自定义指标
- seriesQuery: '{__name__=~"^application.*_requests$",container!="POD",namespace!="",pod!=""}'
resources:
overrides:
namespace: { resource: "namespace" }
pod: { resource: "pod" }
name:
matches: "(.*)_total"
as: "${1}_qps"
metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[30s])) by (<<.GroupBy>>)
检查自定义指标生效情况：
dongjiang@MacBook Pro:echo $ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . | grep "application"
"name": "jobs.batch/application_server_handler_requests",
"name": "namespaces/application_server_request_duration_seconds_bucket",
"name": "namespaces/application_server_response_size_bytes",
"name": "namespaces/application_server_handler_requests_qps",
"name": "namespaces/application_server_request_duration_seconds_sum",
"name": "namespaces/application_server_requests_qps",
"name": "jobs.batch/application_server_request_duration_seconds_sum",
"name": "pods/application_server_response_size_bytes",
"name": "namespaces/application_server_request_duration_seconds_count",
"name": "jobs.batch/application_server_request_size_bytes",
"name": "pods/application_server_requests",
"name": "jobs.batch/application_server_requests",
"name": "services/application_server_request_duration_seconds_sum",
"name": "services/application_server_request_size_bytes",
"name": "pods/application_server_request_duration_seconds_bucket",
"name": "namespaces/application_server_request_size_bytes",
"name": "namespaces/application_server_requests",
"name": "services/application_server_response_size_bytes",
"name": "services/application_server_handler_requests",
"name": "pods/application_server_request_duration_seconds_sum",
"name": "services/application_server_request_duration_seconds_count",
"name": "namespaces/application_server_handler_requests",
"name": "jobs.batch/application_server_request_duration_seconds_bucket",
"name": "services/application_server_request_duration_seconds_bucket",
"name": "jobs.batch/application_server_request_duration_seconds_count",
"name": "pods/application_server_request_size_bytes",
"name": "pods/application_server_requests_qps",
"name": "pods/application_server_handler_requests_qps",
"name": "jobs.batch/application_server_response_size_bytes",
"name": "pods/application_server_request_duration_seconds_count",
"name": "pods/application_server_handler_requests",
"name": "services/application_server_requests",
dongjiang@MacBook Pro:echo $ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq . | grep "qps"
"name": "namespaces/application_server_handler_requests_qps",
"name": "namespaces/application_server_requests_qps",
"name": "pods/application_server_requests_qps",
"name": "pods/application_server_handler_requests_qps",
支持自定义指标监控
更改demo hpa为自定义指标
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2
metadata:
name: echo-custom-metric
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: echo
minReplicas: 1
maxReplicas: 10
behavior:
scaleDown:
stabilizationWindowSeconds: 30
policies:
- type: Percent
value: 100
periodSeconds: 15
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
metrics:
- type: Pods
pods:
metric:
name: application_server_handler_requests
target:
type: AverageValue
averageValue: 200m #0.2
配置生效情况：
dongjiang@MacBook Pro:helm $ kubectl get hpa | grep "custom"
echo-custom-metric Deployment/echo <unknown>/200m 1 10 1 6m46s
dongjiang@MacBook Pro:helm $ kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/application_server_handler_requests' | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "echo-7896cd67b6-gnm7k",
"apiVersion": "/v1"
},
"metricName": "application_server_handler_requests",
"timestamp": "2024-10-28T10:02:43Z",
"value": "233m",
"selector": null
}
]
}
压力情况下
dongjiang@MacBook Pro:helm $ kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/application_server_handler_requests' | jq .
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "echo-7896cd67b6-gnm7k",
"apiVersion": "/v1"
},
"metricName": "application_server_handler_requests",
"timestamp": "2024-10-28T10:07:35Z",
"value": "377m",
"selector": null
},
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "echo-7896cd67b6-pjqc6",
"apiVersion": "/v1"
},
"metricName": "application_server_handler_requests",
"timestamp": "2024-10-28T10:07:35Z",
"value": "53m",
"selector": null
}
]
}
dongjiang@MacBook Pro:helm $ kubectl get hpa | grep "custom"
echo-custom-metric Deployment/echo <unknown>/200m 1 10 2 12m
「如果这篇文章对你有用,请随意打赏」
如果这篇文章对你有用,请随意打赏
使用微信扫描二维码完成支付
