Introduction to Prometheus/Grafana
Prometheus
Prometheus is an open source system for monitoring and alerting. This was initially developed at SoundCloud in 2012. For this system developers of the prometheus takes the inspiration of Googles Borgmon. In Prometheus it uses multidimensional time series data model for metrics and events. Following are the key features of prometheus.
- A multi-dimensional data model with time series data identified by metric name and key/value pairs
- PromQL, a flexible query language to leverage this dimensionality
- no reliance on distributed storage; single server nodes are autonomous
- time series collection happens via a pull model over HTTP
- pushing time series is supported via an intermediary gateway
- targets are discovered via service discovery or static configuration
- multiple modes of graphing and dashboard support
Following are the main components of Prometheus
- Prometheus server
- Client libraries
- Push gateway
- Exporters
- Alertmanager
- various support tools
Following is the architecture of the Prometheus
Grafana
Grafana is an Open Source data visualization tool. We can connect different data sources to grafana and create meaningful, rich dashboard for different workloads. In grafana we can connect different data sources like prometheus, inflexDB, Elasticsearch, Azure logs, collectd.
Installation of Prometheus & Grafana on AKS Cluster
Before we go through the installation of the prometheus following prerequisite must have.
- Azure Kubernetes Cluster
- Azure Account
- Helm Installed in local machine and tiller on cluster
- kubectl
- Ingress controller
For Prometheus installation we can use Helm charts. Its easy to use Helm charts as it configured the deployments.
First we need to clone the git repo of the helm charts
git clone https://github.com/helm/charts.git
Navigate to the folder where prometheus installation code available
cd charts/stable/prometheus
We use following version of prometheus
For installation of prometheus, following helm command can be used
helm install --name=prometheus . --namespace monitoring --set rbac.create=true
Above command will deploy relevant Kubernetes resources that need for prometheus. Such as StatefulSets, Secrets, Deployments, Demonsets, ReplicaSets and Pods. Following is a similar output
helm install --name=prometheus . --namespace monitoring --set rbac.create=true NAME: prometheus LAST DEPLOYED: Thu Mar 14 09:25:12 2019 NAMESPACE: monitoring STATUS: DEPLOYED RESOURCES: ==> v1beta1/ClusterRole NAME AGE prometheus-kube-state-metrics 1s prometheus-server 1s ==> v1/PersistentVolumeClaim NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE prometheus-alertmanager Pending default 1s prometheus-server Pending default 1s ==> v1/ServiceAccount NAME SECRETS AGE prometheus-alertmanager 1 1s prometheus-kube-state-metrics 1 1s prometheus-node-exporter 1 1s prometheus-pushgateway 1 1s prometheus-server 1 1s ==> v1beta1/ClusterRoleBinding NAME AGE prometheus-kube-state-metrics 1s prometheus-server 1s ==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE prometheus-alertmanager ClusterIP 10.0.210.12980/TCP 1s prometheus-kube-state-metrics ClusterIP None 80/TCP 1s prometheus-node-exporter ClusterIP None 9100/TCP 1s prometheus-pushgateway ClusterIP 10.0.188.4 9091/TCP 1s prometheus-server ClusterIP 10.0.226.40 80/TCP 1s ==> v1beta1/DaemonSet NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE prometheus-node-exporter 3 3 0 3 0 1s ==> v1beta1/Deployment NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE prometheus-alertmanager 1 1 1 0 1s prometheus-kube-state-metrics 1 1 1 0 1s prometheus-pushgateway 1 1 1 0 1s prometheus-server 1 1 1 0 1s ==> v1/Pod(related) NAME READY STATUS RESTARTS AGE prometheus-node-exporter-6689b 0/1 ContainerCreating 0 1s prometheus-node-exporter-nszgk 0/1 ContainerCreating 0 1s prometheus-node-exporter-wdw2v 0/1 ContainerCreating 0 1s prometheus-alertmanager-5668757b66-l6hvr 0/2 Pending 0 1s prometheus-kube-state-metrics-56548b5b6-d5t4q 0/1 ContainerCreating 0 1s prometheus-pushgateway-64986d9c4f-jx5c5 0/1 ContainerCreating 0 1s prometheus-server-7df9c555d-7l99z 0/2 Pending 0 0s ==> v1/ConfigMap NAME DATA AGE prometheus-alertmanager 1 1s prometheus-server 3 1s NOTES: The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster: prometheus-server.monitoring.svc.cluster.local Get the Prometheus server URL by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace monitoring port-forward $POD_NAME 9090 The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster: prometheus-alertmanager.monitoring.svc.cluster.local Get the Alertmanager URL by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace monitoring port-forward $POD_NAME 9093 The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster: prometheus-pushgateway.monitoring.svc.cluster.local Get the PushGateway URL by running these commands in the same shell: export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}") kubectl --namespace monitoring port-forward $POD_NAME 9091 For more information on running Prometheus, visit: https://prometheus.io/
After the installation we can use kubectl port forwarding to connect to prometheus UI
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 9090
Following is a full demo of above steps.
Installation of Grafana
For the installation of grafana we use helm as previous. In this we use grafana v6.0.0 as below
Next, change the directory to grafana folder
cd charts/stable/grafana
Next we will use helm command to install grafana to the cluster
helm install stable/grafana --set persistence.enabled=true --set persistence.accessModes={ReadWriteOnce} --set persistence.size=8Gi -n grafana --namespace monitoring
By above helm chart it will create a grafana deployment and other relevant resources. For this I used persistence.enabled=true, persistence.accessModes={ReadWriteOnce} & persistence.size=8Gi. By setting this parameters helm will create persistent volume resources for grafana. If I didn’t provide above parameters, in a case of a node or pod failure the configurations and dashboard configurations will be lost.
helm install stable/grafana --set persistence.enabled=true --set persistence.accessModes={ReadWriteOnce} --set persistence.size=8Gi -n grafana --namespace monitoring
NAME: grafana
LAST DEPLOYED: Thu Mar 14 09:43:08 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/Secret
NAME TYPE DATA AGE
grafana Opaque 3 1s
==> v1/ClusterRoleBinding
NAME AGE
grafana-clusterrolebinding 1s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana ClusterIP 10.0.226.12 80/TCP 1s
==> v1beta1/PodSecurityPolicy
NAME DATA CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
==> v1/ConfigMap
NAME DATA AGE
grafana 1 1s
==> v1/PersistentVolumeClaim
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
grafana Pending default 1s
==> v1/ServiceAccount
NAME SECRETS AGE
grafana 1 1s
==> v1/ClusterRole
NAME AGE
grafana-clusterrole 1s
==> v1beta1/Role
grafana 1s
==> v1beta1/RoleBinding
NAME AGE
grafana 1s
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
grafana 1 1 1 0 1s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
grafana-7b4d7f4469-hxpqq 0/1 Pending 0 1s
NOTES:
1. Get your 'admin' user password by running:
kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:
grafana.monitoring.svc.cluster.local
Get the Grafana URL to visit by running these commands in the same shell:
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=grafana,release=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 3000
3. Login with the password from step 1 and the username: admin
Lets verify the grafana and prometheus are deployed
helm ls --all
After the installation we can login to grafana and perform initial configuration. For login to grafana we need to get the password created by secret. We can run following command
NOTE :- After login change password as for the password policy of organization.
kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Next we can use kubectl port forward to access the grafana UI.
export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=grafana,release=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace monitoring port-forward $POD_NAME 3000
Now all components are install to AKS cluster to up and run the prometheus and grafana. Following is a video for installation of grafana
After installation we need to add data sources and configure the post configuration as follow demo.
Next we can import dashboard from grafana labs or we can create our own dashboard. Following is a demo for import a dashboard from grafana labs.
Thats all for this post, hope this may help for deploying prometheus and grafana to AKS cluster.