Installing and configuring Elastic stack on Ubuntu

Posted on 04th November 2019

The Elastic Stack also known as the ELK stack consists of Elastic Search, Kibana, Beats and Logstash. These products when put together allows you to ingest data that is structured or unstructured, explore and analyse those data and visualize it in real time. This article guides you through the installation and basic configuration of ELK stack on an Ubuntu machine. The same steps could be applied to any Debian based system.

Installing and Configuring Elastic Search

At the core of the Elastic Stack is the Elastic Search, which can store, search, and analyze data that is of any type, whether it is text, numerical, or geospatial.

Follow these steps to install and configure Elastic Search on a Ubuntu or Debian based system.

  1. Download

    Download the Debian package for the latest version of Elasticsearch.

    curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.4.0-amd64.deb
    

    Visit the Elasticsearch downloads page to check the latest version and the download URL for the Deb 64 bit package.

  2. Install

    Run the following command to install the Debian package that you've downloaded.

    sudo dpkg -i elasticsearch-7.4.0-amd64.deb
    
  3. Run

    Configure Elasticsearch service for auto-start.

    sudo systemctl enable elasticsearch.service
    

    Start Elasticsearch service.

    sudo systemctl start elasticsearch.service
    
  4. Verify

    To verify installation is successful, run

    curl http://localhost:9200
    

    which should return something like this:

    {
      "name" : "otg-pc",
      "cluster_name" : "elasticsearch",
      "cluster_uuid" : "eC6pyh_3Sku5AFMEKpgjcw",
      "version" : {
        "number" : "7.4.0",
        "build_flavor" : "default",
        "build_type" : "deb",
        "build_hash" : "22e1767283e61a198cb4db791ea66e3f11ab9910",
        "build_date" : "2019-09-27T08:30:45.569419Z",
        "build_snapshot" : false,
        "lucene_version" : "8.2.0",
        "minimum_wire_compatibility_version" : "6.8.0",
        "minimum_index_compatibility_version" : "6.0.0-beta1"
      },
      "tagline" : "You Know, for Search"
    }
    

Key folders and files

td>Default Data folder.td>/etc/default/elasticsearch
/usr/share/elasticsearchElasticsearch Home folder.
/etc/elasticsearch/Config folder.
/var/log/elasticsearchLog folder.
/var/lib/elasticsearch
/etc/elasticsearch/elasticsearch.ymlConfig file for node and cluster configuration.
Config file for system properties.

Configuring Elasticsearch

Here are some of the basic settings you may be interested in changing. All of these settings discussed below are in the config file /etc/elasticsearch/elasticsearch.yml. You need to have root privilege to edit this file.

  • network_host: IP address to bind to. Default is 127.0.0.1 and [::1]. If you want Elastic search service to listen to all IP address on your machine then set this to 0.0.0.0

  • http.port: HTTP port to bind. Default is 9200.

  • cluster.name: Name for your Elasticsearch cluster. Default name is elasticsearch.

  • node.name: Name for the node. Default is the host name.

  • path.data: Path to the data directory. Default is /var/lib/elasticsearch

  • path.logs: Path to the log files directory. Default is /var/log/elasticsearch

  • discovery.seed_hosts: List of other nodes in the cluster that are master-eligible and can be used to seed the discovery process. The defaulis 127.0.0.1, [::1].

  • cluster.initial_master_nodes: List of nodes whose votes will be used for electing a cluster master initially. You must use the same name that you use in node.name setting.

Installing and configuring Kibana

The job for Kibana in the Elastic stack is to visualize the data. It acts like a front-end for Elasticsearch. Follow these steps to install and configure Kibana on Ubuntu and other Debian-based systems.

  • Download

    Download the Kibana Debian package.

    curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-7.4.0-amd64.deb
    

    The download may take few minutes as it is quite big in size. Visit Kibana Downloads page to check the latest version and get the download URL to use the curl command above.

  • Install

    To install Kibana from the debian package, run

    sudo dpkg -i kibana-7.4.0-amd64.deb
    
  • Run

    Configure Kibana to start automatically on boot.

    sudo systemctl enable kibana.service
    

    Start Kibana service.

    sudo systemctl start kibana.service
    
  • Verify

    To verify the installation, point your web browser at http://localhost:5601

    which should display the Kibana home page.

    Kibana home

    To display the status page, go to the url http://localhost:5601/status

Install Logstash (optional)

Logstash allows you to process and ingest the data in to Elasticsearch. You can modify the data whichever way you like before it is pushed to an index in Elasticsearch. Follow these steps to install and configure Logstash on Ubuntu or Debian based system.

Download

Download the Debian package for the latest version of Logstash.

curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-7.4.1.deb

Install

Before installing Logstash you need to install Java.

sudo apt-get install openjdk-8-jre
sudo dpkg -i logstash-7.4.1.deb

Run

To enable and run Logstash as a service, run the following commands.

sudo systemctl enable logstash.service
sudo systemctl start logstash.service

Verify

To verify the logstash installation we can create and run a basic pipeline. The following command reads string from the standard input, converts to uppercase and display on standart ouput.

cd /usr/share/logstash/
bin/logstash -e 'input { stdin { } } filter { output { stdout { }}'

After you run the above command, enter some message in the terminal, say for example, you enter "hello world" and logstash will output the "HELLO WORLD" along with timestamp, version and host.

hello world
{
      "@version" => "1",
       "message" => "HELLO WORLD",
    "@timestamp" => 2019-10-29T18:07:59.678Z,
          "host" => "otg"
}

Install Beats (optional)

Once the installation of Elastic stack is complete, you may install one or more Beats which allows you to capture data from different sources. Visit Beats download" page to pick your beat.

Here are the commands to install Filebeat for example.

Download

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.4.1-amd64.deb

Install

sudo dpkg -i filebeat-7.4.1-amd64.deb

Run

To run filebeat as a service:

sudo systemctl enable filebeat.service
sudo systemctl start filebeat.service

Verify

To verify the install, we can configure filebeat to send logs directly to elasticsearch. Edit the config file /etc/filebeat/filebeat.yml and set the following options.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

setup.dashboards.enabled: true

output.elasticsearch:
  hosts: ["localhost:9200"]

You need to restart filebeat service after changing the configuration.

sudo systemctl restart filebeat.service

Finally to see the logs, open the Kibana webpage (http://localhost:5601), then go to Discover page and select filebeat-* on the index list.

Filebeat on Kibana

Post a comment

Comments

Nothing yet..be the first to share wisdom.