Log Aggregation With Grafana+Loki+Promtail

Grafana Loki is a set of components that can be composed into a fully featured logging stack.

Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the file system. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.

Promtail: It is an agent which ships the contents of local logs to a private Loki instance . It is usually deployed to every machine that has applications needed to be monitored.

I am using the Loki Data Source with the Promtail client as the collector on the Grafana Server itself.

Steps we are going to perform in this article:

• Install and configure Loki Binary.
• Install and configure Promtail Binary.
• Configure Loki Data Source and Grafana.

Step 1: Install and configure Loki Binary.

Go to directory

cd /usr/local/bin

Download the latest version of Loki from its official page.

sudo curl -O -L "https://github.com/grafana/loki/releases/download/v1.4.1/loki-linux-amd64.zip"

Unzip the Loki zip file

 sudo unzip loki-linux-amd64.zip

Allow the executable permission to the Loki binary.

 sudo chmod +x loki-linux-amd64

Create a Loki config file

sudo vi config-loki.yml

Add this file in config-loki.yml

[pastacode lang=”bash” manual=”auth_enabled%3A%20false%20%0Aserver%3A%20http_listen_port%3A%203100%20%0Aingester%3A%20%0Alifecycler%3A%20%0Aaddress%3A%20127.0.0.1%20%0Aring%3A%20%0Akvstore%3A%0A%20store%3A%20%0Ainmemory%20replication_factor%3A%201%20%0Afinal_sleep%3A%200s%20%0Achunk_idle_period%3A%205m%20%0Achunk_retain_period%3A%2030s%20%0Aschema_config%3A%0A%20configs%3A%20%0A-%20from%3A%202018-04-15%20%0Astore%3A%20boltdb%0A%20object_store%3A%20filesystem%20%0Aschema%3A%20v9%20%0Aindex%3A%0A%20prefix%3A%20index_%20%0Aperiod%3A%20168h%20storage%0Astorage_config%3A%20%0Aboltdb%3A%20%0Adirectory%3A%20%2Ftmp%2Floki%2Findex%20%0Afilesystem%3A%20%0Adirectory%3A%20%2Ftmp%2Floki%2Fchunks%20%0Alimits_config%3A%0Aenforce_metric_name%3A%20false%20%0Areject_old_samples%3A%20true%20%0Areject_old_samples_max_age%3A%20168h%20chunk_store_config%3A%20%0Amax_look_back_period%3A%200%20%0Atable_manager%3A%0Achunk_tables_provisioning%3A%0A%20inactive_read_throughput%3A%200%0A%20inactive_write_throughput%3A%200%0A%20provisioned_read_throughput%3A%200%0A%20provisioned_write_throughput%3A%200%0A%20index_tables_provisioning%3A%0A%20inactive_read_throughput%3A%200%0A%20inactive_write_throughput%3A%200%0A%20provisioned_read_throughput%3A%200%0A%20provisioned_write_throughput%3A%200%20retention_deletes_enabled%3A%20false%0A%20retention_period%3A%200″ message=”” highlight=”” provider=”manual”/]

save and exit

Now Configure Loki as a service

sudo vi /etc/systemd/system/loki.service

Add the following

[Unit]
Description=Loki service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/loki-linux-amd64 -config.file /usr/local/bin/config-loki.yml

[Install]
WantedBy=multi-user.target

save and exit

Start the service

systemctl start loki.service
systemctl enable loki.service
systemctl status loki.service

Open the browser and check if the URL is working or not

http://<IP_Address:3100/metrics>

its working

Step 2: Install and configure Promtail Binary

In this article i am install and configure in other linux server .

go to directory

cd /usr/local/bin

Download the latest version of Promtail agent from official page.

sudo curl -O -L "https://github.com/grafana/loki/releases/download/v0.4.0/promtail-linux-amd64.gz"

unzip the promtail zip file

sudo gunzip promtail-linux-amd64.gz

Allow the executable permission to the Promtail binary.

sudo chmod +x promtail-linux-amd64

Create a promtail config file.

sudo vi config-promtail.yml

Add the following in config-promtail.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://52.66.213.126:3100/loki/api/v1/push

scrape_configs:

  - job_name: system
    entry_parser: raw
    static_configs:
    - targets:
        - localhost
      labels:
        job: varlogs
        host: ip-172-31-43-74.ap-south-1.compute.internal
        __path__: /var/log/*log

save and exit

Create a service file for promtail

sudo vi /etc/systemd/system/promtail.service

Add the following script

[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file /usr/local/bin/config-promtail.yml

[Install]
WantedBy=multi-user.target

exit and save

Start and check the service

sudo systemctl start promtail.service
sudo systemctl enable promtail.service
sudo systemctl status promtail.service

Open the browser and check if the URL is working or not

http://<ip_address>:9080

its working

Step 3: Configure Loki Data Source and Grafana.

For Grafana Installation please refer to this post:

Grafana Installation

Add data source in grafana

Save and test

Open Explorer

Drop down the Log labels tab and select file name/host/job as we defined in the promtail.yml file .

After selecting the filename you will able to see the logs

You can also search the logs according to your requirement.

Congratulations… You follow all the steps and configured log aggregation tool

Thank you…