Application Performance Monitoring: Monitor dynamically java applications with Consul, Prometheus and Grafana

1- Introduction

2- Versions

3- Code

4- Check the monitoring stack

5- Start an application instance

6- The metrics

7- Consul registration

8- Prometheus scraping

9- Grafana dashboard

10- Interesting links

11- Conclusion

Introduction

The goal of this article is to detail with a real example how Consul, Prometheus and Grafana can be leveraged in order to monitor java applications dynamically.
This java application is having JMX and Micrometer metrics.

Article overview

Versions

• Java 11
• Spring Boot 2.6.6
• Spring Cloud 2021.0.0
• Consul 1.11.1
• Prometheus 2.24.1
• Grafana 7.3.7

Code

Monitoring stack

Install Docker, clone the project, then run the following:

cd docker-apm-showroom/consul-prometheus-grafana
docker-compose up

nbodev / docker-apm-showroom

Application to monitor

Clone the project, then run the following in order to compile:

cd springboot-apm-showroom
mvn clean install

nbodev / springboot-apm-showroom

Check the monitoring stack

Visit the following urls and check if your monitoring stack is up and running:

• Consul: http://localhost:8500/
• Prometheus: http://localhost:9092/
• Grafana: http://localhost:3000/

Start an application instance

Run the following in order to start an instance of the application, we assume you cloned the project under /workspace/springboot-apm-showroom:

java -javaagent:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx_prometheus_javaagent-0.16.1.jar=7771:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx-exporter-config.yml -Djmx.export.port=7771 -Dnode.name=node1 -Dserver.port=9991 -jar ./target/apm-showroom-0.0.1-SNAPSHOT.jar

Explanation of jvm arguments :

• JMX exporter: this is the Prometheus JMX exporter agent that will expose the JMX metrics of the application so Prometheus can scrape them.
• We need to specify the port of this agent. Note that this is different from the Spring Boot application port, it is just another port.

-javaagent:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx_prometheus_javaagent-0.16.1.jar=<jmx exporter port>

• JMX exporter port: this argument is proper to my application, you will see that this will be used later on to register the JMX exporter instance to Consul (spoiler alert).

-Djmx.export.port=<jmx exporter port>

• Node name: this argument is proper to my application, it is used to distinguish the different instances.

-Dnode.name=<node name>

• Spring Boot application port: this is the classic Spring Boot port argument.

-Dserver.port=<Spring Boot application port>

Make sure that you change the following in case you start more than one instance:

java -javaagent:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx_prometheus_javaagent-0.16.1.jar=<jmx exporter port>:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx-exporter-config.yml -Djmx.export.port=<jmx exporter port> 
-Dnode.name=<node name> 
-Dserver.port=<Spring Boot application port> 
-jar ./target/apm-showroom-0.0.1-SNAPSHOT.jar

Let’s dive into the details …

The metrics

The metrics are defined in two classes: DemoJMX and DemoMicrometer.

DemoJMX defines the metrics as MBeans (demoLastInt, demoLastDouble), you can see them in the jconsole for instance.

DemoMicrometer defines the metrics as Micrometer metrics (demoCounter, demoGauge), we expose them to Prometheus thanks to this section that is added in the pom.xml:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

Note that those metrics (JMX and Micrometer) keep changing randomly every … milliseconds, a method with this annotation @Scheduled(fixedRate = ...) is used in each class.

Consul registration

Spring Cloud

Spring Cloud is used in order to register to Consul, for that we need to add those dependencies in the pom.xml:

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    
    ...
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
    ...	
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-consul-discovery</artifactId>
        </dependency>

In the application.yml we have this configuration:

# enable all the actuator endpoints
management:
  endpoints:
    web:
      exposure:
        include: "*"
        
# spring cloud configuration, exposes the micrometer metrics and registers to consul
spring:
  application:
    name: my-service
  cloud:
    consul:
      host: localhost
      port: 8500
      discovery:
        # instance-id must not be empty, must start with letter, end with a letter or digit, and have as interior characters only letter, digits, and hyphen
        instance-id: ${spring.application.name}-${node.name}-micrometer-instance
        serviceName: ${spring.application.name}
        register: true
      retry: 
        initial-interval: 2000
        max-attempts: 10000
        max-interval: 4000
        multiplier: 1.1

• The first instance that will register to Consul thanks to Spring Cloud is the instance that will expose the Micrometer metrics. This instance is running on the port of Spring Boot (e.g. 9991), assuming I run my application with those jvm arguments:

java -javaagent:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx_prometheus_javaagent-0.16.1.jar=7771:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx-exporter-config.yml -Djmx.export.port=7771 -Dnode.name=node1 -Dserver.port=9991 -jar ./target/apm-showroom-0.0.1-SNAPSHOT.jar

• The instance name that will be seen on Consul side will be my-service-node1-micrometer-instance and the metrics are available at this url http://localhost:9991/actuator/prometheus (in Prometheus format).

• Note that we are setting up a retry configuration here in case Consul is unavailable, see the retry section in the application.yml, this requires the following to be added in your pom.xml:

<!-- Spring Retry in order to retry the connection if Consul is unavailable -->
        <dependency>
            <groupId>org.springframework.retry</groupId>
            <artifactId>spring-retry</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>

Custom-made class ConsulServiceRegisterConfig

• Note that we also have the JMX exporter in the jvm arguments, this will also be registered as another instance to Consul, this instance has this name my-service-node1-jmx-instance according to our configuration.
• The JMX metrics are available here http://localhost:7771/actuator/prometheus, the port 7771 is the one used for my JMX exporter.

• This instance registers to Consul thanks to the custom-made class com.nbodev.apm.consul.ConsulServiceRegisterConfig that uses Consul API. There is a configuration in the application.yml that applies to this class, note that we use the jmx.export.port in this configuration.

# bespoke consul registration of jmx exporter into consul, exposes the jmx metrics
apm:
  consul:
    host: ${spring.cloud.consul.host}
    port: ${spring.cloud.consul.port}
  node:
    healthcheck:
      port: ${jmx.export.port}
      path: actuator/prometheus
    service:
      name: ${spring.application.name}
    instance:
      name: ${spring.application.name}-${node.name}-jmx-instance

Results after registration to Consul of the two instances related to node1

At this stage we have two instances related to the same service my-service, those instances registered themselves to Consul.

Prometheus scraping

Prometheus does not need to know the port of the JMX exporter and relies on the fact that the same service name my-service is used for the two instances (Micrometer and JMX exporter) when we register them to Consul.

Prometheus will use consul_sd_configs with the myservice name as defined by the services when we registered them.

See Prometheus configuration below, we focus on the service and scrape /actuator/prometheus.

This sub url is already the metrics path for the Micrometer metrics (http://localhost:9991/actuator/prometheus) and for the JMX exporter metrics (http://localhost:7771/actuator/prometheus).

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

scrape_configs:
# consul job
- job_name: 'consul'
  consul_sd_configs:
  - server: 'apm-showroom-consul-server:8500'
    services:
    - my-service
  metrics_path: '/actuator/prometheus'
  relabel_configs:
  - source_labels: [__meta_consul_service_id]
    regex: 'my-service-(.*)-(.*)-(.*)'
    replacement: 'my-service-$1'
    target_label: node
  - source_labels: [__meta_consul_service_id]
    target_label: instance
# prometheus job
- job_name: 'prometheus'
  static_configs:
  - targets: 
    - 'apm-showroom-prometheus:9090'

Prometheus scrape all the instances related to the service my-service by resolving them dynamically.

No need to know the ip address nor the hostname nor the port.

You can see in the prometheus.yml configuration that we create a new label node thanks to the relabel_config this label will have the node name without the reference to the metrics type.

Note that the prometheus.yml file is available in the https://gitlab.com/nbodev/docker-apm-showroom project, under consul-prometheus-grafana/prometheus/config/prometheus.yml.

e.g. the JMX instance will have this instance label instance="my-service-node1-jmx-instance", the Micrometer instance will have this instance label instance="my-service-node1-micrometer-instance".
After relabeling, we create a new label node with this value node="my-service-node1". This is achieved with this part of theprometheus.yml configuration.

relabel_configs:
  - source_labels: [__meta_consul_service_id]
    regex: 'my-service-(.*)-(.*)-(.*)'
    replacement: 'my-service-$1'
    target_label: node

This will be used as a variable in Grafana, so we can filter on a particular node in order to get the related metrics.

Prometheus detects the two instances and can scrape them now

Let’s say we start a new node with the following jvm arguments:

java -javaagent:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx_prometheus_javaagent-0.16.1.jar=7772:/workspace/springboot-apm-showroom/src/main/resources/jmx-exporter-config/jmx-exporter-config.yml -Djmx.export.port=7772 -Dnode.name=node1 -Dserver.port=9992 -jar ./target/apm-showroom-0.0.1-SNAPSHOT.jar

The node2 underlying instances register to Consul:

Prometheus detects them:

Prometheus detects the two new instances and can scrape the node1 and node2 now

Grafana dashboard

Go to http://localhost:3000/.

Login with admin/admin.

Add the Prometheus data source, use http://apm-showroom-prometheus:9090 as url.

Import the dashboard located in consul-prometheus-grafana/grafana/dashboard/APM Showroom.json from the https://gitlab.com/nbodev/docker-apm-showroom project.

Make sure that you have at least one Spring Boot application running, you will see the following when you pick up the APM Showroomdashboard.

You can see the dashboard for all nodes:

APM Showroon dashboard — All nodes

You can filter on one node:

APM Showroon dashboard — filter on node1

This dashboard has a variable named node defined with the Prometheus up metrics (part of Prometheus built in metrics) and the Regex /.*node="(.*)"/ , this node that you see in the Regex is the one that has been created with the Prometheus relabeling

node variable

Prometheus up metric

Then for each widget, you need to use this variable, see below for instance, we define the metrics that will be seen with demo_gauge{node=~"$node"} .

In short we get the demo_gauge metric for the $node variable, we filter on the variable value.

Interesting links

Micrometer

• https://micrometer.io/docs/concepts
• https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-metrics
• https://github.com/prometheus/jmx_exporter

Spring Cloud

• https://docs.spring.io/spring-cloud-consul/docs/current/reference/html/index.html
• https://cloud.spring.io/spring-cloud-consul/reference/html/appendix.html

Consul

• https://learn.hashicorp.com/tutorials/consul/docker-container-agents
• https://www.consul.io/docs/agent
• https://github.com/Ecwid/consul-api

Prometheus

• https://prometheus.io/docs/prometheus/latest/querying/basics/
• https://prometheus.io/docs/prometheus/latest/configuration/configuration/
• https://www.fosstechnix.com/prometheus-promql-tutorial-with-examples/

Grafana

• https://grafana.com/docs/grafana/latest/datasources/prometheus/
• https://grafana.com/docs/grafana/latest/dashboards/export-import/
• https://grafana.com/docs/grafana/latest/variables/

Conclusion

Hope you got a glimpse of how we can monitor a java application in a dynamic way with Consul, Prometheus and Grafana.

Note that I did not insist on some concepts related to Prometheus or Grafana as you can refer to the official documentation or online tutorials related to those products.