Collecting Server Stats with Prometheus, Node-Exporter and Grafana

On my server, I run Prometheus, Node Exporter, and Grafana using Portainer. Here’s some info on my setup below.

Prometheus

Prometheus is an open-source monitoring and alerting toolkit originally developed at SoundCloud and now part of the Cloud Native Computing Foundation (CNCF). It is designed to monitor services and applications, collect metrics, and generate alerts based on predefined conditions.

Key Features:

1. Time Series Data: Prometheus collects metrics as time series data, which means it stores data with a timestamp, a metric name, and key-value pairs (labels) to organize the data.
2. Multi-dimensional Data Model: It uses labels (key-value pairs) to identify metrics and filter data efficiently.
3. PromQL: A powerful query language called PromQL (Prometheus Query Language) is used to extract and work with collected data.
4. Pull Model: Prometheus typically uses a pull model to scrape (fetch) metrics from applications via HTTP endpoints, rather than a push model.
5. Service Discovery: It can automatically discover services via integrations with platforms like Kubernetes, EC2, and others, simplifying large-scale deployments.
6. Alerting: Prometheus integrates with Alertmanager to send alerts to various channels like email, Slack, PagerDuty, etc.
7. Storage: Prometheus stores time series data on disk in a highly efficient format. For long-term storage, it can be integrated with other storage systems.

Prometheus is widely used in cloud-native and microservices environments to monitor the health and performance of applications and infrastructure. It excels in scenarios requiring real-time monitoring and dynamic scaling, especially when combined with tools like Grafana for visualization.

Node-Exporter

Node Exporter is a specialized tool used in conjunction with Prometheus to collect and expose hardware and operating system-level metrics from a server. It’s an agent that runs on each node (or server) and provides detailed metrics about the system’s performance, such as CPU usage, memory usage, disk I/O, and network statistics, which Prometheus can then scrape and store.

Key Features:

1. System Metrics: Node Exporter gathers a wide range of metrics related to the operating system, such as:
   • CPU and memory usage
   • Disk and filesystem statistics
   • Network traffic and statistics
   • System load and uptime
   • Temperature and power metrics (if supported by the hardware)
2. Exported as HTTP: Metrics are made available through an HTTP endpoint (typically at http://<host>:9100/metrics) that Prometheus can scrape.
3. Designed for Linux: Although it is primarily used on Linux systems, it also has partial support for other operating systems like Windows and BSD.
4. Modular Metrics: Metrics are organized into various subsystems (e.g., CPU, memory, disk), and users can enable or disable specific collectors depending on what they want to monitor.
5. Lightweight: Node Exporter is designed to have a minimal impact on system resources, making it suitable for production environments.
6. Open Metrics Format: Metrics are provided in a standardized format that Prometheus understands.

Common Use Cases:

• Monitoring Resource Utilization: It helps track CPU, memory, and disk usage trends to ensure efficient resource allocation.
• System Health Checks: Node Exporter provides critical insights into server performance for troubleshooting and alerting on infrastructure issues.
• Capacity Planning: By analyzing trends in resource usage, Node Exporter aids in capacity planning for scaling up or optimizing infrastructure.

Together with Prometheus, Node Exporter is a key part of monitoring infrastructure in cloud-native and containerized environments like Kubernetes.

Grafana

Grafana is an open-source data visualization and monitoring platform that allows users to create and share interactive dashboards. It is designed to work with a variety of data sources, enabling users to visualize and analyze metrics in real-time. Grafana is commonly used alongside monitoring tools like Prometheus, but it supports a wide range of databases and time-series data sources.

Key Features:

1. Data Visualization: Grafana provides rich, interactive visualizations, including graphs, charts, heatmaps, tables, and more. Users can create dynamic, real-time dashboards that offer deep insights into their metrics.
2. Multiple Data Sources: Grafana supports many popular data sources out of the box, including:
   • Prometheus
   • InfluxDB
   • Elasticsearch
   • MySQL, PostgreSQL
   • Loki (Grafana’s log aggregation system)
   • Cloud-based services (AWS CloudWatch, Google Cloud Monitoring, etc.)
3. Custom Dashboards: Users can build highly customizable dashboards with drag-and-drop panels. Dashboards can display data from multiple sources, combining them into a single view for better understanding of overall system performance.
4. Alerting: Grafana includes built-in alerting features. Users can set up alerts based on thresholds for metrics, and Grafana will send notifications through various channels like email, Slack, or PagerDuty.
5. Templating: Dashboards can be templated, allowing users to create reusable dashboards that change dynamically based on inputs, like time ranges, server names, or environments (e.g., staging, production).
6. Annotations: Users can mark important events on graphs to provide context (e.g., when a deployment happened or an issue occurred), making it easier to correlate system changes with metrics.
7. User Management: Grafana supports user roles and permissions, enabling teams to collaborate securely by controlling access to specific dashboards and features.
8. Plugins: It has a vibrant ecosystem of plugins, allowing users to extend Grafana’s functionality with additional visualizations, data source integrations, and tools.
9. Multi-Platform Support: Grafana can run on various operating systems and in containerized environments, making it flexible for different infrastructures.

Common Use Cases:

• Infrastructure Monitoring: Grafana is frequently used to monitor servers, services, and applications, especially in combination with Prometheus, to provide insights into system performance.
• Business Analytics: It is used to create dashboards that track business metrics, such as website performance, customer usage, or sales data, making it valuable beyond IT infrastructure.
• Incident Response: By integrating alerts and visualizations, Grafana is often used in incident response workflows, helping teams identify and resolve issues quickly.

Overall, Grafana is a powerful tool for monitoring, alerting, and gaining real-time insights into system and business metrics through visual dashboards.

Direct / Full Installation

Personally, I think it’s way simpler to setup all 3 services: Grafana, Prometheus and Node-Exporter using Docker-Compose and Portainer. If you haven’t used Portainer before, please checkout this page

To run Prometheus, Node Exporter, and Grafana with Docker Compose, you’ll need to define a docker-compose.yml file that configures the services and their dependencies. Here’s how you can set it up:

Step-by-Step Docker Compose Setup

1. Directory Structure:

You will need to create a directory for the Docker Compose project and create configuration files for Prometheus. The structure could look like this:

monitoring/
│
├── prometheus/
│   ├── prometheus.yml
│
└── docker-compose.yml

2. Prometheus Configuration (prometheus/prometheus.yml):

Create a Prometheus configuration file to define the scraping jobs (including scraping Node Exporter).

Example:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['prometheus:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

This is my file:

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'node'
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['10.0.0.111']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['10.0.0.111:9100']

3. Docker Compose File (docker-compose.yml):

In the root monitoring directory, create the docker-compose.yml file to configure the services.

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    ports:
      - "9100:9100"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_SECURITY_ADMIN_USER=admin
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

volumes:
  grafana_data:

Breakdown of the Docker Compose Setup:

• Prometheus:
  • Runs the prom/prometheus:latest Docker image.
  • Uses the Prometheus configuration file (prometheus.yml) from the local directory to configure scraping jobs.
  • Exposes Prometheus on port 9090.
• Node Exporter:
  • Runs the prom/node-exporter:latest Docker image.
  • Exposes metrics on port 9100.
• Grafana:
  • Runs the grafana/grafana:latest Docker image.
  • Exposes Grafana on port 3000.
  • Uses grafana_data volume to persist data across container restarts.
  • Admin username and password are set to admin by default for simplicity (you can modify these values).

4. Running the Stack:

Navigate to the directory where your docker-compose.yml file is located and run the following command:

docker-compose up -d

This will start the Prometheus, Node Exporter, and Grafana containers.

5. Accessing the Services:

• Prometheus: Access the Prometheus UI at http://<server-ip>:9090
• Node Exporter: Metrics from Node Exporter will be available at http://<server-ip>:9100/metrics, and Prometheus will scrape these metrics automatically.
• Grafana: Access the Grafana UI at http://<server-ip>:3000. Log in with the default credentials (admin/admin).

6. Configuring Grafana:

Once logged into Grafana, you need to add Prometheus as a data source:

• Go to Configuration → Data Sources.
• Select Prometheus and set the URL to http://prometheus:9090.
• Save the configuration, and you can start creating dashboards using Prometheus data.

This setup will provide a complete monitoring solution with Prometheus scraping system metrics from Node Exporter and Grafana for visualization.

Portainer Setup

To set up Prometheus, Node Exporter, and Grafana using Portainer, you’ll need to follow a similar approach as with Docker Compose, but through Portainer’s user interface. Portainer provides an easy way to manage Docker containers, images, and networks through a graphical interface, including deploying applications with docker-compose.

Here’s how to deploy Prometheus, Node Exporter, and Grafana on Portainer:

Step-by-Step Guide for Using Portainer

1. Access Portainer:

First, log in to your Portainer instance. If you haven’t installed Portainer yet, you can do so using Docker:

docker volume create portainer_data
docker run -d -p 9000:9000 -p 9443:9443 --name=portainer \
    --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    portainer/portainer-ce:latest

Once installed, access the Portainer UI by navigating to http://<your-server-ip>:9000.

2. Create a New Stack (Docker Compose Equivalent):

Portainer allows you to deploy containers using Docker Compose stacks directly from the interface.

1. Log into Portainer.
2. In the left-hand menu, select Stacks.
3. Click on Add stack.
4. Enter a name for the stack (e.g., monitoring-stack).

3. Use the Docker Compose Configuration:

Copy and paste the following docker-compose.yml into the Web editor in Portainer:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - prometheus_data:/etc/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    ports:
      - "9100:9100"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_SECURITY_ADMIN_USER=admin
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

volumes:
  grafana_data:
  prometheus_data:

4. Create a Prometheus Configuration File:

For Prometheus to work correctly, you need to create the configuration file prometheus.yml on the host machine or directly inside Portainer.

If you’re working from the command line:

1. On the Docker host, create the prometheus.yml file:

   mkdir -p ~/portainer_stacks/monitoring
   nano ~/portainer_stacks/monitoring/prometheus.yml
   

2. Add the following configuration to prometheus.yml:

   global:
     scrape_interval: 15s
   
   scrape_configs:
     - job_name: 'prometheus'
       static_configs:
         - targets: ['localhost:9090']
   
     - job_name: 'node-exporter'
       static_configs:
         - targets: ['node-exporter:9100']
   

3. Save the file.

Alternatively, if you’re editing from Portainer, you can attach the file manually by specifying the file’s path inside Portainer or use a bind mount in the Stack configuration.

5. Deploy the Stack:

After pasting the docker-compose.yml file into Portainer:

1. Click on Deploy the stack.
2. Portainer will now pull the necessary Docker images (Prometheus, Grafana, and Node Exporter) and set up the services based on the provided configuration.

6. Verify the Services:

Once the stack is deployed, you can verify that everything is running properly.

• Prometheus: Access Prometheus at http://<your-server-ip>:9090.
• Node Exporter: Prometheus will automatically scrape Node Exporter metrics at http://<your-server-ip>:9100/metrics.
• Grafana: Access Grafana at http://<your-server-ip>:3000 (default login: admin/admin).

7. Configure Grafana:

Once logged into Grafana, add Prometheus as a data source:

• Go to Configuration → Data Sources.
• Select Prometheus and set the URL to http://prometheus:9090.
• Click Save & Test.

The stack will allow you to monitor system metrics and visualize them in Grafana. You can modify and expand this setup based on your specific monitoring requirements.

Direct Installation without Docker

To install Grafana, Prometheus, and Node Exporter on a vanilla Ubuntu Server 22.04, you will need to install several packages. Here’s a step-by-step breakdown of the installation process for each component:

1. Prometheus Installation

Prometheus is not available in the default Ubuntu repositories, so you will need to download it from the official Prometheus website.

Steps:

1. Update the system:

   sudo apt update
   sudo apt upgrade -y
   

2. Create Prometheus user and directories:

   sudo useradd --no-create-home --shell /bin/false prometheus
   sudo mkdir /etc/prometheus
   sudo mkdir /var/lib/prometheus
   

3. Download Prometheus:

   wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz
   tar -xvf prometheus-2.46.0.linux-amd64.tar.gz
   cd prometheus-2.46.0.linux-amd64
   sudo cp prometheus /usr/local/bin/
   sudo cp promtool /usr/local/bin/
   sudo cp -r consoles /etc/prometheus/
   sudo cp -r console_libraries /etc/prometheus/
   sudo cp prometheus.yml /etc/prometheus/
   

4. Create a systemd service for Prometheus:

   sudo nano /etc/systemd/system/prometheus.service
   

   Paste the following content:

   [Unit]
   Description=Prometheus
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=prometheus
   Group=prometheus
   Type=simple
   ExecStart=/usr/local/bin/prometheus \
      --config.file=/etc/prometheus/prometheus.yml \
      --storage.tsdb.path=/var/lib/prometheus/ \
      --web.console.templates=/etc/prometheus/consoles \
      --web.console.libraries=/etc/prometheus/console_libraries
   
   [Install]
   WantedBy=multi-user.target
   

5. Enable and start Prometheus:

   sudo systemctl daemon-reload
   sudo systemctl enable prometheus
   sudo systemctl start prometheus
   

2. Node Exporter Installation

Node Exporter is responsible for gathering system-level metrics.

Steps:

1. Create a user for Node Exporter:

   sudo useradd --no-create-home --shell /bin/false node_exporter
   

2. Download Node Exporter:

   wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
   tar -xvf node_exporter-1.6.1.linux-amd64.tar.gz
   sudo cp node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
   

3. Create a systemd service for Node Exporter:

   sudo nano /etc/systemd/system/node_exporter.service
   

   Paste the following content:

   [Unit]
   Description=Node Exporter
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=node_exporter
   Group=node_exporter
   Type=simple
   ExecStart=/usr/local/bin/node_exporter
   
   [Install]
   WantedBy=multi-user.target
   

4. Enable and start Node Exporter:

   sudo systemctl daemon-reload
   sudo systemctl enable node_exporter
   sudo systemctl start node_exporter
   

3. Grafana Installation

Grafana can be installed via its official repository for Ubuntu.

Steps:

1. Add the Grafana repository:

   sudo apt install -y software-properties-common
   sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
   

2. Install the Grafana GPG key:

   sudo wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
   

3. Install Grafana:

   sudo apt update
   sudo apt install grafana
   

4. Enable and start Grafana:

   sudo systemctl enable grafana-server
   sudo systemctl start grafana-server
   

Verifying the Installations:

• Prometheus: Access Prometheus at http://<server-ip>:9090
• Node Exporter: Access Node Exporter metrics at http://<server-ip>:9100/metrics
• Grafana: Access Grafana at http://<server-ip>:3000 (default login is admin/admin)

What’s Next ?

Checkout these:

• Grafana 11 deep dive
• Demo: Explore Metrics

About Guillaume Plante

A developper with a passion for technology, music, astronomy and art. Coding range: hardware/drivers, security, ai,. c/c++, powershell

Email : guillaumeplante.qc@gmail.com

Website : https://arsscriptum.ddns.net