Introduction:
Keeping an eye on your Kubernetes cluster can feel like a daunting task, but with the right tools, it doesn’t have to be. Prometheus and Grafana – two fantastic open-source tools that make monitoring and visualizing your cluster easy.
Lets look into how you can use Prometheus and Grafana to keep tabs on your Kubernetes cluster. Lets look into how you can set up, the basics of configuring them.
Setup suing helm:
Will be using helm to install both these tools to avoid writing multiple yamls for various components.
Get familiar with helm, checkout this artcle → Helm — Your Compass in the Kubernetes Universe
Lets roll.
First add Prometheus to the repo and update the helm repository.
Install Prometheus with the command below.
helm install prometheus prometheus-community/prometheus
We can use kubectl get all command to list all the resources and see all the pods, svcs and deployment belongings to prom.
To access the prom web ui, expose the service named “Prometheus-server”.
With the newly exposed service we have an external/public IP which we can use to get the prom web ui.
In the Status → targets, we see all the endpoints where Prometheus is getting the metrics from.
Install Grafana:
Similarly how we did for prom, we add the grafana repo and update it.
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install grafana grafana/grafana
Expose grafana service to be able to access the web interface.
The default username is admin, and to get the password, use kubectl get secret command.
kctl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
You can change the password if you want after login.
Click on Data sources and select Prometheus, fill out the connection details. Click save and test the connection.
Now I will create a dashboard. You can create your own or import existing dashboards from grafana. For this example, I’m going to import a few.
From the Prometheus home page, click on Dashboards, choose import and use 315 or similar.
Once it’s added, we should be able to view the data.
Kube-state matrics:
Kube state metrics service provides many metrics for deployments, pods, jobs, cronjobs etc. which is not available by default.The service has been created however we need to expose it and change Prometheus config.
kctl expose service prometheus-kube-state-metrics --type=NodePort --target-port=8080 --name=prom-kstate-ext
Add the metrics end point as a scrape config job in the Prometheus config.
You can add even more dashboards for different metrics. Example: 6417 for pod metrics
Wrap Up:
There we have it. Monitoring K8s with prom and grafana. Next experiment is to configure alert manager for email alerts.