How to remove/deregister an instance from PMM?

We all need a monitoring system like pmm to monitor our database. Let me give a brief about how pmm works.


So PMM uses prometheus to store metrics/graphs and grafana to display them. As seen in the architecture diagram prometheus uses consul to get the list of host to scrap metrics. More information on architecture and installation are available in docs.
We were facing issues related to down hosts and if setup alerts in grafana for down host and host is terminated it continuously throws alerts. So problem was how can we remove/deregister instances from pmm.
We can access consul UI using below link:
http://PMM-SERVER/consul/#/dc1/
Previously there was option available on this ui to deregister host. while in the new releases of consul it has been removed see link.
But now we can use below procedure from command line to deregister hosts from consul.
Get list of nodes:
curl -s 'http://USER:PASSWORD@PMM-SERVER-IP/v1/internal/ui/nodes?dc=dc1' | python -mjson.tool | less
Remove node from consul:
curl -s -X PUT -d '{"Datacenter":"dc1","Node":"NODE-NAME"}' 'http://USER:PASSWORD@PMM-SERVER-IP/v1/catalog/deregister?dc=dc1'
After that, it is needed to remove data from Prometheus:
curl -X DELETE 'http://USER:PASSWORD@PMM-SERVER-IP/prometheus/api/v1/series?match\[\]=\{instance="NODE-NAME"\}'
Kindly make a note node names in consul and in prometheus might be different for the one host
Percona is also planning to add Deregister button on interface in PMM-544.

Comments