Kubernetes - Prometheus

 Prometheus  Prometheus is an open-source monitoring framework. It provides out-of-the-box monitoring capabilities for the Kubernetes contai...


 Prometheus is an open-source monitoring framework. It provides out-of-the-box monitoring capabilities for the Kubernetes container orchestration platform.

Metric Collection: Prometheus uses the pull model to retrieve metrics over HTTP.

Metric Endpoint: The systems that you want to monitor using Prometheus should expose the metrics on an /metrics endpoint. Prometheus uses this endpoint to pull the metrics in regular intervals

PromQL: Prometheus comes with PromQL, a very flexible query language that can be used to query the metrics in the Prometheus dashboard. Also, the PromQL query will be used by Prometheus UI and Grafana to visualize metrics.

Prometheus Exporters: Exporters are libraries that convert existing metrics from third-party apps to Prometheus metrics format. 

TSDB (time-series database): Prometheus uses TSDB for storing all the data. By default, all the data gets stored locally. However, there are options to integrate remote storage for Prometheus TSDB.

Prometheus Monitoring Setup on Kubernetes

I assume that you have a Kubernetes cluster up and running with kubectl setup on your workstation.

Connect to your Kubernetes cluster and make sure you are having admin privileges. 

Step 1:

First, we will create a Kubernetes namespace for all our monitoring components. If you don’t create a dedicated namespace, all the Prometheus kubernetes deployment objects get deployed on the default namespace.

           kubectl create namespace monitoring

Step 2:

Prometheus uses Kubernetes APIs to read all the available metrics from Nodes, Pods, Deployments, etc. For this reason, we need to create an RBAC policy with read access to required API groups and bind the policy to the monitoring namespace.

Create a file clusterRole.yaml

kind: ClusterRole
  name: prometheus
- apiGroups: [""]
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
kind: ClusterRoleBinding
  name: prometheus
  kind: ClusterRole
  name: prometheus
- kind: ServiceAccount
  name: default
  namespace: monitoring

Execute the file which creates role and role binding on cluster level for default service account in monitoring namespace.

    kubectl create -f clusterRole.yaml

Step 3:

Create a Config Map To Externalize Prometheus Configurations

All configurations for Prometheus are part of prometheus.yaml file and all the alert rules for Alertmanager are configured in prometheus.rules.

prometheus.yaml: This is the main Prometheus configuration which holds all the scrape configs, service discovery details, storage locations, data retention configs, etc)

prometheus.rules: This file contains all the Prometheus alerting rules

By externalizing Prometheus configs to a Kubernetes config map, you don’t have to build the Prometheus image whenever you need to add or remove a configuration. You need to update the config map and restart the Prometheus pods to apply the new configuration.

The config map with all the Prometheus scrape config and alerting rules gets mounted to the Prometheus container in /etc/prometheus location as prometheus.yaml and prometheus.rules files.

 Create a file called config-map.yaml and copy the file contents from here Prometheus Config file

 Execute the following command to create the config map in Kubernetes.

     kubectl create -f config-map.yaml

Step 4:

Create a Prometheus Deployment

 Create a file named prometheus-deployment.yaml and copy the following contents onto the file. In this configuration, we are mounting the Prometheus config map as a file inside /etc/prometheus

apiVersion: apps/v1
kind: Deployment
  name: prometheus-deployment
  namespace: monitoring
    app: prometheus-server
  replicas: 1
      app: prometheus-server
        app: prometheus-server
        - name: prometheus
          image: prom/prometheus
            - "--storage.tsdb.retention.time=12h"
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - containerPort: 9090
              cpu: 500m
              memory: 500M
              cpu: 1
              memory: 1Gi
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
        - name: prometheus-config-volume
            defaultMode: 420
            name: prometheus-server-conf
        - name: prometheus-storage-volume
          emptyDir: {}

Execute above deployment

   kubectl create  -f prometheus-deployment.yaml 

You can check the created deployment using the following command.

   kubectl get deployments --namespace=monitoring

Step 5:

Access the deployment by using NodePort service

 Create a file named prometheus-service.yaml and copy the following contents. We will expose Prometheus on all Kubernetes node IPs on port 30000.

apiVersion: v1
kind: Service
  name: prometheus-service
  namespace: monitoring
  annotations: 'true'   '9090'
    app: prometheus-server
  type: NodePort  
    - port: 8080
      targetPort: 9090
      nodePort: 30000

Execute this service to Access Prometheus Dashboard

    kubectl create -f prometheus-service.yaml --namespace=monitoring

Step 6: 

Goto browser and access the Prometheus Dashboard on port number 30000



Ansible,6,AWS,1,Azure DevOps,1,Containerization with docker,2,DevOps,2,Docker Quiz,1,Docker Swarm,1,DockerCompose,1,ELK,2,git,2,Jira,1,Kubernetes,1,Kubernetes Quiz,5,SAST DAST Security Testing,1,SonarQube,3,Splunk,2,vagrant kubernetes,1,YAML Basics,1,
DevOpsWorld: Kubernetes - Prometheus
Kubernetes - Prometheus
Loaded All Posts Not found any posts VIEW ALL Readmore Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS PREMIUM CONTENT IS LOCKED STEP 1: Share to a social network STEP 2: Click the link on your social network Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy Table of Content