Kubernetes - Prometheus

 Prometheus  Prometheus is an open-source monitoring framework. It provides out-of-the-box monitoring capabilities for the Kubernetes contai...

 Prometheus

 Prometheus is an open-source monitoring framework. It provides out-of-the-box monitoring capabilities for the Kubernetes container orchestration platform.

Metric Collection: Prometheus uses the pull model to retrieve metrics over HTTP.

Metric Endpoint: The systems that you want to monitor using Prometheus should expose the metrics on an /metrics endpoint. Prometheus uses this endpoint to pull the metrics in regular intervals

PromQL: Prometheus comes with PromQL, a very flexible query language that can be used to query the metrics in the Prometheus dashboard. Also, the PromQL query will be used by Prometheus UI and Grafana to visualize metrics.

Prometheus Exporters: Exporters are libraries that convert existing metrics from third-party apps to Prometheus metrics format. 

TSDB (time-series database): Prometheus uses TSDB for storing all the data. By default, all the data gets stored locally. However, there are options to integrate remote storage for Prometheus TSDB.

Prometheus Monitoring Setup on Kubernetes

I assume that you have a Kubernetes cluster up and running with kubectl setup on your workstation.

Connect to your Kubernetes cluster and make sure you are having admin privileges. 

Step 1:

First, we will create a Kubernetes namespace for all our monitoring components. If you don’t create a dedicated namespace, all the Prometheus kubernetes deployment objects get deployed on the default namespace.

           kubectl create namespace monitoring

Step 2:

Prometheus uses Kubernetes APIs to read all the available metrics from Nodes, Pods, Deployments, etc. For this reason, we need to create an RBAC policy with read access to required API groups and bind the policy to the monitoring namespace.

Create a file clusterRole.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  name: prometheus

rules:

- apiGroups: [""]

  resources:

  - nodes

  - nodes/proxy

  - services

  - endpoints

  - pods

  verbs: ["get", "list", "watch"]

- apiGroups:

  - extensions

  resources:

  - ingresses

  verbs: ["get", "list", "watch"]

- nonResourceURLs: ["/metrics"]

  verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: prometheus

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: prometheus

subjects:

- kind: ServiceAccount

  name: default

  namespace: monitoring

Execute the file which creates role and role binding on cluster level for default service account in monitoring namespace.

    kubectl create -f clusterRole.yaml

Step 3:

Create a Config Map To Externalize Prometheus Configurations

All configurations for Prometheus are part of prometheus.yaml file and all the alert rules for Alertmanager are configured in prometheus.rules.

prometheus.yaml: This is the main Prometheus configuration which holds all the scrape configs, service discovery details, storage locations, data retention configs, etc)

prometheus.rules: This file contains all the Prometheus alerting rules

By externalizing Prometheus configs to a Kubernetes config map, you don’t have to build the Prometheus image whenever you need to add or remove a configuration. You need to update the config map and restart the Prometheus pods to apply the new configuration.

The config map with all the Prometheus scrape config and alerting rules gets mounted to the Prometheus container in /etc/prometheus location as prometheus.yaml and prometheus.rules files.

 Create a file called config-map.yaml and copy the file contents from here Prometheus Config file

 Execute the following command to create the config map in Kubernetes.

     kubectl create -f config-map.yaml

Step 4:

Create a Prometheus Deployment

 Create a file named prometheus-deployment.yaml and copy the following contents onto the file. In this configuration, we are mounting the Prometheus config map as a file inside /etc/prometheus

apiVersion: apps/v1

kind: Deployment

metadata:

  name: prometheus-deployment

  namespace: monitoring

  labels:

    app: prometheus-server

spec:

  replicas: 1

  selector:

    matchLabels:

      app: prometheus-server

  template:

    metadata:

      labels:

        app: prometheus-server

    spec:

      containers:

        - name: prometheus

          image: prom/prometheus

          args:

            - "--storage.tsdb.retention.time=12h"

            - "--config.file=/etc/prometheus/prometheus.yml"

            - "--storage.tsdb.path=/prometheus/"

          ports:

            - containerPort: 9090

          resources:

            requests:

              cpu: 500m

              memory: 500M

            limits:

              cpu: 1

              memory: 1Gi

          volumeMounts:

            - name: prometheus-config-volume

              mountPath: /etc/prometheus/

            - name: prometheus-storage-volume

              mountPath: /prometheus/

      volumes:

        - name: prometheus-config-volume

          configMap:

            defaultMode: 420

            name: prometheus-server-conf

 

        - name: prometheus-storage-volume

          emptyDir: {}

Execute above deployment

   kubectl create  -f prometheus-deployment.yaml 

You can check the created deployment using the following command.

   kubectl get deployments --namespace=monitoring

Step 5:

Access the deployment by using NodePort service

 Create a file named prometheus-service.yaml and copy the following contents. We will expose Prometheus on all Kubernetes node IPs on port 30000.

apiVersion: v1

kind: Service

metadata:

  name: prometheus-service

  namespace: monitoring

  annotations:

      prometheus.io/scrape: 'true'

      prometheus.io/port:   '9090'

spec:

  selector:

    app: prometheus-server

  type: NodePort  

  ports:

    - port: 8080

      targetPort: 9090

      nodePort: 30000

Execute this service to Access Prometheus Dashboard

    kubectl create -f prometheus-service.yaml --namespace=monitoring

Step 6: 

Goto browser and access the Prometheus Dashboard on port number 30000