Skip to content

Installation and Configuration Guide for Promeheus

Prometheus is a monitoring system that collects and stores data.

This setup guide is for the Prometheus server and should be done only once on the machine collecting all metrics.

For all VMs you want to monitor, follow the client setup guide found here

Setup Main Server Guide

Setup on the 'main' monitoring server.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# Create the Prometheus & Node-exporter Users
sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

sudo groupadd --system node_exporter
sudo useradd -s /sbin/nologin --system -g node_exporter node_exporter

# verify users
getent passwd prometheus
getent passwd node_exporter

# install 
sudo apt update

cd /tmp

wget https://github.com/prometheus/prometheus/releases/download/v3.1.0/prometheus-3.1.0.linux-amd64.tar.gz
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz

tar -xvf prometheus-3.1.0.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz

sudo mv prometheus-3.1.0.linux-amd64/prometheus /usr/bin/
sudo mv prometheus-3.1.0.linux-amd64/promtool /usr/bin/
sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/bin/prometheus-node-exporter

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo mkdir /usr/share/prometheus

sudo chown prometheus:prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/share/prometheus
sudo chown prometheus:prometheus /usr/bin/prometheus
sudo chown prometheus:prometheus /usr/bin/promtool

# Verify Services
sudo systemctl is-enabled prometheus
sudo systemctl status prometheus
sudo systemctl is-enabled prometheus-node-exporter
sudo systemctl status prometheus-node-exporter

Configure

Run: sudo vim /etc/prometheus/prometheus.yml and insert:

Change the script values

Only need to change the HOSTNAME

/etc/prometheus/prometheus.yml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: '$HOSTNAME' # CHANGE-ME: Uniquely identifies this instance

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['127.0.0.1:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'node'
    # If prometheus-node-exporter is installed, grab stats about the local
    # machine by default.
    static_configs:
      - targets: ['127.0.0.1:9100']

Now, run: sudo vim /lib/systemd/system/prometheus.service and insert:

/lib/systemd/system/prometheus.service
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
[Unit]
Description=Monitoring system and time series database
Documentation=https://prometheus.io/docs/introduction/overview/ man:prometheus(1)
Requires=network-online.target
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
WorkingDirectory=/usr/share/prometheus
EnvironmentFile=-/etc/conf.d/prometheus
ExecStart=/usr/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/data  \
  --web.listen-address="127.0.0.1:9090"
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

# systemd hardening-options
AmbientCapabilities=
CapabilityBoundingSet=
DeviceAllow=/dev/null rw
DevicePolicy=strict
LimitMEMLOCK=0
LimitNOFILE=32768
LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
PrivateUsers=true
ProtectControlGroups=true
ProtectHome=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=full
RemoveIPC=true
RestrictNamespaces=true
RestrictRealtime=true
SystemCallArchitectures=native
SyslogIdentifier=prometheus

[Install]
WantedBy=multi-user.target
/lib/systemd/system/prometheus.service
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
[Unit]
Description=Monitoring system and time series database
Documentation=https://prometheus.io/docs/introduction/overview/ man:prometheus(1)
Requires=network-online.target
After=network-online.target nss-lookup.target
Wants=network-online.target nss-lookup.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
WorkingDirectory=/usr/share/prometheus
EnvironmentFile=-/etc/conf.d/prometheus
ExecStart=/usr/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/data  \
  --web.listen-address="127.0.0.1:9090" \
  --web.enable-remote-write-receiver
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

# systemd hardening-options
AmbientCapabilities=
CapabilityBoundingSet=
DeviceAllow=/dev/null rw
DevicePolicy=strict
LimitMEMLOCK=0
LimitNOFILE=32768
LockPersonality=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
PrivateUsers=true
ProtectControlGroups=true
ProtectHome=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=full
RemoveIPC=true
RestrictNamespaces=true
RestrictRealtime=true
SystemCallArchitectures=native
SyslogIdentifier=prometheus

[Install]
WantedBy=multi-user.target

Run: sudo vim /lib/systemd/system/prometheus-node-exporter.service and insert:

/lib/systemd/system/prometheus-node-exporter.service
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
[Unit]
Description=Prometheus exporter for machine metrics
Requires=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Restart=on-failure
EnvironmentFile=-/etc/conf.d/prometheus-node-exporter
ExecStart=/usr/bin/prometheus-node-exporter --web.listen-address="127.0.0.1:9100" 
NoNewPrivileges=true
ProtectHome=read-only
ProtectSystem=strict

[Install]
WantedBy=multi-user.target

Enable services

# Restart and enable
sudo systemctl daemon-reload

sudo systemctl restart prometheus
sudo systemctl enable prometheus
sudo systemctl status prometheus

sudo systemctl restart prometheus-node-exporter
sudo systemctl enable prometheus-node-exporter
sudo systemctl status prometheus-node-exporter

Setup Nginx

# Install
sudo apt install -y nginx

Run: sudo vim /etc/nginx/conf.d/prometheus.conf and insert:

Change the script values

Replace [$SERVER_IP] with the IP on this VM.

/etc/nginx/conf.d/prometheus.conf
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# Configure Nginx as Reverse Proxy

server {
    listen 8080;
    server_name [$SERVER_IP]; # CHANGEME

    location / {
      add_header X-Frame-Options DENY;
      add_header X-Content-Type-Options nosniff;
      add_header X-XSS-Protection "1; mode=block";
      add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

      proxy_pass http://127.0.0.1:9090;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;
    }
}

Verify Installation

# Test and restart
sudo nginx -t
sudo systemctl restart nginx

Access Web Interface

Prometheus:

http://server_ip

Grafana:

http://server_ip:3000

Next Steps

Now, check out the client guide to monitor other VM's.

To visualize the metrics, Grafana is recommended and guide is found here