Прикладной мониторинг

Для управления мониторингом в кластерах платформы Штурвал используется Prometheus Operator. Данный контроллер поддерживает следующие пользовательские ресурсы (Kubernetes Custom Resources):

Для установки прикладного мониторинга необходимо создать пользовательский ресурс Prometheus, с одной из приведённых далее конфигураций. За сбор метрик отвечают ресурсы PodMonitor и ServiceMonitor.

Подготовка к установке прикладного мониторинга #

Неймспейсы, в которые устанавливаются приложения, требующие сбор метрик, должны быть отмечены лейблом shturval.tech/user-monitoring
Для этого запустите в cli команду:

$ kubectl create namespace test-app
namespace/test-app created
$ kubectl label namespace test-app shturval.tech/user-monitoring=
namespace/test-app labeled

Или примените манифест:

---
apiVersion: v1
kind: Namespace
metadata:
  name: test-app
  shturval.tech/user-monitoring: ''

Установка прикладного мониторинга #

Создайте манифест в формате yaml с листингом прикладного контура.

Пример:

---
apiVersion: v1
kind: Namespace
metadata:
  name: user-monitoring
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: user-monitoring
  namespace: user-monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: user-monitoring
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: user-monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: user-monitoring
subjects:
- kind: ServiceAccount
  name: user-monitoring
  namespace: user-monitoring
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: user-monitoring
  name: user-monitoring
  namespace: user-monitoring
spec:
  enableAdminAPI: false
  image: r.shturval.tech/prometheus/prometheus:v2.42.0
  listenLocal: false
  logFormat: logfmt
  logLevel: info
  paused: false
  podMetadata:
    labels:
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/instance: user-monitoring
  externalLabels:
    cluster: test-cluster
  # remoteWrite:
  # - tlsConfig:
  #     insecureSkipVerify: true
  #   url: http://URL_ВНЕШНЕГО_PROMETHEUS
  # Для перенаправления логов снимите комментирование 
  replicas: 1
  resources:
    limits:
      memory: 2Gi
    requests:
      cpu: 200m
      memory: 512Mi
  retention: 14d
  retentionSize: 14GB
  routePrefix: /
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  serviceAccountName: user-monitoring
  serviceMonitorNamespaceSelector:
    matchExpressions:
    - key: shturval.tech/platform-monitoring
      operator: DoesNotExist
    - key: shturval.tech/user-monitoring
      operator: Exists
  serviceMonitorSelector:
    matchExpressions:
    - key: release
      operator: NotIn
      values:
      - "shturval-metrics"
  podMonitorNamespaceSelector:
    matchExpressions:
    - key: shturval.tech/platform-monitoring
      operator: DoesNotExist
    - key: shturval.tech/user-monitoring
      operator: Exists
  podMonitorSelector:
    matchExpressions:
    - key: release
      operator: NotIn
      values:
      - "shturval-metrics"
  probeNamespaceSelector:
    matchExpressions:
    - key: shturval.tech/platform-monitoring
      operator: DoesNotExist
    - key: shturval.tech/user-monitoring
      operator: Exists
  probeSelector:
    matchExpressions:
    - key: release
      operator: NotIn
      values:
      - "shturval-metrics"
  ruleNamespaceSelector:
    matchExpressions:
    - key: shturval.tech/platform-monitoring
      operator: DoesNotExist
    - key: shturval.tech/user-monitoring
      operator: Exists
  ruleSelector:
    matchExpressions:
    - key: release
      operator: NotIn
      values:
      - "shturval-metrics"
  shards: 1
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 18Gi
        storageClassName: local-path
  version: v2.42.0
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: user-monitoring
  name: prometheus-user-monitoring
  namespace: user-monitoring
spec:
  ports:
  - name: web
    port: 9090
    targetPort: web
  selector:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: user-monitoring
  type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/instance: user-monitoring
    release: shturval-metrics
  name: prometheus-user-monitoring
  namespace: user-monitoring
spec:
  endpoints:
    - port: web
  namespaceSelector:
    matchNames:
      - user-monitoring
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/instance: user-monitoring

Команда для установки #

Для установки прикладного мониторинга запустите в cli команду:

$ kubectl apply -f НАЗВАНИЕ_СОЗДАННОГО_МАНИФЕСТА.yaml 
namespace/user-monitoring created
serviceaccount/user-monitoring created
clusterrole.rbac.authorization.k8s.io/user-monitoring created
clusterrolebinding.rbac.authorization.k8s.io/user-monitoring created
prometheus.monitoring.coreos.com/user-monitoring created
service/prometheus-user-monitoring created
servicemonitor.monitoring.coreos.com/prometheus-user-monitoring created