Monitoring Board Install and Deploy

From the Apache IoTDB 1.0 version, we introduced the system monitoring module, you can complete the Apache IoTDB important operational indicators for monitoring, this article describes how to open the system monitoring module in the Apache IoTDB distribution, and the use of Prometheus + Grafana way to complete the visualisation of the system monitoring indicators.

pre-preparation

software requirement

1. Apache IoTDB: version 1.0 and above, download from the official website: https://iotdb.apache.org/Download/
2. Prometheus: version 2.30.3 and above, download from the official website: https://prometheus.io/download/
3. Grafana: version 8.4.2 and above, download from the official website: https://grafana.com/grafana/download
4. IoTDB-Grafana installer: Grafana Dashboards is an TimechoDB(Enterprise Edition based on IoTDB) tool, and you may contact your sales for the relevant installer.

cluster requirement

Make sure that the IoTDB cluster is started before doing the following.

clarification

This doc will build the monitoring dashboard on one machine (1 ConfigNode and 1 DataNode) environment, other cluster configurations are similar, users can adjust the configuration according to their own cluster situation (the number of ConfigNode and DataNode). The basic configuration information of the cluster built in this paper is shown in the table below.

NODETYPE	NODEIP	Monitor Pusher	Monitor Level	Monitor Port
ConfigNode	127.0.0.1	PROMETHEUS	IMPORTANT	9091
DataNode	127.0.0.1	PROMETHEUS	IMPORTANT	9093

configure Prometheus capture monitoring metrics

1. Download the installation package. Download the Prometheus binary package locally, unzip it and go to the corresponding folder:

tar xvfz prometheus-*.tar.gz
cd prometheus-*

2. Modify the configuration. Modify the Prometheus configuration file prometheus.yml as follows:
   a. Added confignode task to collect monitoring data from ConfigNode
   b. Add datanode task to collect monitoring data from DataNode

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
    - targets: ["localhost:9090"]
  - job_name: "confignode"
    static_configs:
    - targets: ["localhost:9091"]
    honor_labels: true
  - job_name: "datanode"
    static_configs:
    - targets: ["localhost:9093"]
    honor_labels: true

3. Start Promethues. the default expiration time for Prometheus monitoring data is 15d. in production environments, it is recommended to adjust the expiration time to 180d or more in order to track historical monitoring data for a longer period of time, as shown in the following startup command:

./prometheus --config.file=prometheus.yml --storage.tsdb.retention.time=180d

4. Confirm the startup is successful. Enter http://localhost:9090 in the browser to enter Prometheus, click to enter the Target interface under Status (Figure 1 below), when you see State are Up, it means the configuration is successful and connected (Figure 2 below), click the link on the left side to jump to the webpage monitoring.

Using Grafana to View Monitoring Data

Step1：Grafana Installation, Configuration and Startup

1. Download the binary package of Grafana locally, unzip it and go to the corresponding folder:

tar -zxvf grafana-*.tar.gz
cd grafana-*

2. Start Grafana and enter:

./bin/grafana-server web 

3. Enter http://localhost:3000 in your browser to access Grafana, the default initial username and password are both admin.
4. First we configure the Data Source in Configuration to be Prometheus.

5. When configuring the Data Source, pay attention to the URL where Prometheus is located, and click Save & Test after configuration, the Data source is working prompt appears, then the configuration is successful.

Step2：Use the official Grafana dashboard provided by IoTDB

1. Enter Grafana，click Browse of Dashboards

2. Click the Import button on the right

3. Select a way to import Dashboard
   a. Upload the Json file of the downloaded Dashboard locally
   b. Enter the URL or ID of the Dashboard obtained from the Grafana website
   c. Paste the contents of the Dashboard's Json file

4. Select Prometheus in the Dashboard as the Data Source you just configured and click Import

5. Then enter Dashboard，select job to be ConfigNode，then following monitoring dashboard will be seen:

6. Similarly, we can import the Apache DataNode Dashboard, select job as DataNode,then following monitoring dashboard will be seen:

Step3：Creating a new Dashboard for data visualisation

1. First create the Dashboard, then create the Panel.

2. After that, you can visualize the monitoring-related data in the panel according to your needs (all relevant monitoring metrics can be filtered by selecting confignode/datanode in the job first).

3. Once the visualisation of the monitoring metrics selected for attention is complete, we get a panel like this: