Gather Infrastructure Metrics with Topbeat and ELK on CentOS 7

Introduction

Topbeat, which is one of the several “Beats” data shippers that helps send various types of server data to an Elasticsearch instance, allows you to gather information about the CPU, memory, and process activity on your servers. In conjunction with an ELK server (Elasticsearch, Logstash, and Kibana), the data that Topbeat gathers can be used to easily visualize metrics so that you can see the status of your servers in a centralized place.

In this tutorial, we will show you how to use an ELK stack to gather and visualize infrastructure metrics by using Topbeat on a CentOS 7 server.

 

Load Topbeat Index Template in Elasticsearch

Because we are planning on using Topbeat to ship logs to Elasticsearch, we should load the Topbeat index template. The index template will configure Elasticsearch to analyze incoming Topbeat fields in an intelligent way.

First, download the Topbeat index template on your ELK Server:

cd /usr/src
curl -O https://raw.githubusercontent.com/elastic/topbeat/master/etc/topbeat.template.json

Then load the template with this command:

curl -XPUT 'http://localhost:9200/_template/topbeat' -d@topbeat.template.json

Now your ELK server is ready to accept data from Topbeat. Let’s set up Topbeat on a client server next.

Set Up Topbeat (Add Client Servers)

Do these steps for each CentOS or Red Hat-based server that you want to send metrics data to Logstash on your ELK Server.

Copy SSL Certificate (if you don’t have it)

On your ELK Server, copy the SSL certificate to your Client Server (substitute the client server’s address, and your own login):

scp /etc/pki/tls/certs/logstash-forwarder.crt user@client_server_private_address:/tmp

After providing your login’s credentials, ensure that the certificate copy was successful. It is required for communication between the client servers and the ELK Server.

Now, on your Client Server, copy the ELK Server’s SSL certificate into the appropriate location (/etc/pki/tls/certs):

mkdir -p /etc/pki/tls/certs
cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/

Now we will install the Topbeat package.

Install Topbeat Package

On Client Server, create run the following command to import the Elasticsearch public GPG key into rpm (if you don’t have it):

rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch

Create and edit a new yum repository file for Filebeat:

echo '[beats]
name=Elastic Beats Repository
baseurl=https://packages.elastic.co/beats/yum/el/$basearch
enabled=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
gpgcheck=1
' | tee /etc/yum.repos.d/elastic-beats.repo

Save and exit.

Install Topbeat with this command:

yum -y install topbeat

Topbeat is now installed but not yet configured.

Configure Topbeat

Now we will configure Topbeat to connect to Logstash on our ELK Server. This section will step you through modifying the example configuration file that comes with Topbeat.

On Client Server, create and edit Topbeat configuration file:

vi /etc/topbeat/topbeat.yml

Note: Topbeat’s configuration file is in YAML format, which means that indentation is very important! Be sure to use the same number of spaces that are indicated in these instructions.

Near the top of the file, you will see the input section, which is where you can specify which metrics and statistics should be sent to the ELK server. We’ll use the default input settings, but feel free to change it to fit your needs.

Under the output section, find the line that says elasticsearch:, which indicates the Elasticsearch output section (which we are not going to use). Delete or comment out the entire Elasticsearch output section (up to the line that says #logstash:).

Find the commented out Logstash output section, indicated by the line that says #logstash:, and uncomment it by deleting the preceding #. In this section, uncomment the hosts: ["localhost:5044"] line. Change localhost to the private IP address (or hostname, if you went with that option) of your ELK server:

  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["ELK_server_private_IP:5044"]

This configures Topbeat to connect to Logstash on your ELK Server at port 5044 (the port that we specified a Logstash input for in the prerequisite tutorial).

Next, find the tls section, and uncomment it. Then uncomment the line that specifies certificate_authorities, and change its value to ["/etc/pki/tls/certs/logstash-forwarder.crt"]. It should look something like this:

...
    tls:
      # List of root certificates for HTTPS server verifications
      certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

This configures Topbeat to use the SSL certificate that we created on the ELK Server in the prerequisite tutorial.

Save and quit.

Now restart Topbeat to put our changes into place:

systemctl restart topbeat
systemctl enable topbeat

Now Topbeat is sending your client server’s system, processes, and filesystem metrics to your ELK server! Repeat this section for all of the other servers that you wish to Topbeat metrics for.

Test Topbeat Installation

If your ELK stack is setup properly, Topbeat (on your client server) should be shipping your logs to Logstash on your ELK server. Logstash should be loading the Topbeat data into Elasticsearch in an date-stamped index, topbeat-YYYY.MM.DD.

On your ELK Server, verify that Elasticsearch is indeed receiving the data by querying for the Topbeat index with this command:

curl -XGET 'http://localhost:9200/topbeat-*/_search?pretty'

You should see a bunch of output that looks like this:

Sample Output:
{
      "_index" : "topbeat-2016.02.01",
      "_type" : "process",
      "_id" : "AVKeLSdP4HKUFv4CjZ7K",
      "_score" : 1.0,
      "_source":{"@timestamp":"2016-02-01T18:51:43.937Z","beat":{"hostname":"topbeat-01","name":"topbeat-01"},"count":1,"proc":{"cpu":{"user":0,"user_p":0,"system":50,"total":50,"start_time":"12:54"},"mem":{"size":0,"rss":0,"rss_p":0,"share":0},"name":"jbd2/vda1-8","pid":125,"ppid":2,"state":"sleeping"},"type":"process","@version":"1","host":"topbeat-01"}
}

If your output shows 0 total hits, Elasticsearch is not loading any Topbeat data under the index you searched for, and you should review your setup for errors. If you received the expected output, continue to the next step.

 

Connect to Kibana

When you are finished setting up Topbeat on all of the servers that you want to gather logs for, let’s look at Kibana, the web interface that we installed earlier.

In a web browser, go to the FQDN or public IP address of your ELK Server. After entering the “kibanaadmin” credentials, you should see a page prompting you to configure a default index pattern:

File_Beat_Default_01

Go ahead and select topbeat-* from the Index Patterns menu (left side), then click the Star (Set as default index) button to set the Topbeat index as the default.

Now click the Discover link in the top navigation bar. By default, this will show you all of the log data over the last 15 minutes. You should see a histogram with log events, with log messages below:

Kibana_Discover

Reference Links:

Print Friendly, PDF & Email

Pablo Javier Furnari

Linux System Administrator at La Plata Linux
I'm a Linux Sysadmin with 8 years of experience. I work with several clients as a consulter here in Argentina and oversea (I have clients in the United States, Mexico, Pakistan and Germany).

I know my strengths and weaknesses. I'm a quick learner, I know how to work with small and big teams. I'm hard worker, proactive and I achieve everything I propose.

Leave a Reply

Your email address will not be published. Required fields are marked *


CAPTCHA Image
Reload Image