Running ELK stack on docker - full solution

06 Jun 2016

If you’ve read my Measure, Monitor, Observe and supervise post, you know I am quite the freak of monitoring and logging everything in the system.

For logging purposes, the ELK stack is by far the best solution out there, and I have
tried a lot of them, from SAAS to self hosted ones.

However, from a Devops standpoint, ELK can be quite difficult to install, whether as a distributed solution or on a single machine.

I open sourced the way Gogobot is doing the logging with Rails over a year ago, in the blog post Parsing and centralizing logs from Ruby/Rails and nginx - Our Recipe.

This solution and a version of this chef recipe is running at Gogobot until this very day but I wanted to make it better.

Making it better doesn’t only mean running the latest versions of the stack, it also means having an easier way to run the stack locally and check for problems, but it also means making it more portable.

The moving parts

Before diving deeper into the solution lets first sketch out what are the moving parts of an ELK stack, including one part that is often overlooked.

ELK Stack

Going a bit into the roles here

Nginx - Providing a proxy into Kibana and authentication layer on top
Logstash - Parsing incoming logs and inserting the data into Elasticsearch
Kibana - Providing visualization and exploration tools above Elasticsearch
ElasticSearch - Storage, index and search solution for the entire stack.

Running in dev/production

Prerequisites

If you want to follow this blog post writing the commands and actually having a solution running you will need the following

virtualbox
docker-machine
docker-compose

Docker is a great way to run this stack in dev/production. In dev we use docker-compose and in production we use chef to orchestrate and provision the containers on top of EC2.

Dockers

Nginx

First, lets create the Docker for nginx.

$ mkdir -p ~/Code/kibana-nginx
$ cd ~/Code/kibana-nginx

We will need an htpasswd, this file will contain the username and password combination that users will be required to use in order to view Kibana.

You can use this generator online or any other solution you see fit.

Create a file called kibana.htpasswd in the same directory and paste the content in.

For example:

kibana:$apr1$Z/5.LALa$P0hfDGzGNt8VtiumKMyo/0

Now, our nginx will need a configuration to use, so lets create that now

Create a file called nginx.conf in the same directory

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    access_log    /var/log/nginx/access.log;

    include       /etc/nginx/conf.d/*.conf;
    include       /etc/nginx/sites-enabled/*;
}

And now, lets create a file called kibana.conf that will be our “website” on nginx.

server {
  listen 80 default_server;
  server_name logs.avitzurel.com;
  location / {
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/conf.d/kibana.htpasswd;
    proxy_pass http://kibana:5601;
  }
}

Now, we will need the Dockerfile, which looks like this:

FROM nginx
COPY kibana.htpasswd /etc/nginx/conf.d/kibana.htpasswd
COPY nginx.conf /etc/nginx/nginx.conf
COPY kibana.conf /etc/nginx/sites-enabled/kibana.conf

As you can see, we are using all the files we’ve created earlier.

You will need to build the docker (and eventually push it when going beyond dev). For the purpose of this post lets assume it’s kensodev/kibana-nginx, you can obviously rename it to whatever you want.

$ docker build -t kensodev/kibana-nginx
$ docker push kensodev/kibana-nginx

Logstash

Before I dive into the logstash configuration, I want to emphasize how we ship logs to logstash.

All logs are shipped to logstash through syslog, we use native syslog without any modification. All machines write simple log files and syslog monitors it and sends it to logstash via TCP/UDP. There is no application specific shipper or any other solution.

Diving in

I like creating my own image for logstash as well, gives me more control over what volumes I want to use, copying patterns over and more. So, lets do this now.

$ mkdir -p docker-logstash
$ cd docker-logstash

Here’s the Dockerfile, This is a simplified version of what I am running in production but it will do for this blog post. (Feel free to comment/ask questions below if something is unclear)

FROM logstash:latest
COPY logstash/config/nginx-syslog.conf /opt/logstash/server/etc/conf.d/nginx-syslog
EXPOSE 5000
CMD ["logstash"]

Few parts we’ll notice here.

I am exposing port 5000 for this example, in real life I am exposing more ports as I need them.

I have a single configuration file called nginx-syslog.conf here again, in real life I have about 5 per logstash instance. I try to keep my log types simple, makes life much easier.

nginx-syslog.conf

input {
  tcp {
    port => "5000"
    type => "syslog"
  }
  udp {
    port => "5000"
    type => "syslog"
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
  }
}

filter {
  if [type] == 'syslog' {
    date {
      match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
      remove_field => [ "timestamp" ]
    }

    useragent {
      source => "agent"
    }

    mutate {
      convert => ["response", "integer"]
      convert => ["bytes", "integer"]
      convert => ["responsetime", "float"]
    }

    geoip {
      source => "clientip"
      target => "geoip"
      add_tag => [ "nginx-geoip" ]
    }

    grok {
      match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
      overwrite => [ "message" ]
    }
  }
}

Now, we will build the docker image

$ docker build -t kensodev/logstash
$ docker push kensodev/logstash

Moving on to composing the solution

Now that we have our custom docker images, lets move over to composing the solution together using docker-compose

Keep in mind here, so far we are working locally, you don’t have to docker-push for any of this to work on your local machine, compose will default to the local image if you have it

Create a docker-compose.yml file and paste in this content

nginx:
  image: kensodev/kibana-nginx
  links:
    - kibana
  ports:
    - "80:80"
elasticsearch:
  image: elasticsearch:latest
  command: elasticsearch -Des.network.host=0.0.0.0
  ports:
    - "9200:9200"
    - "9300:9300"
logstash:
  command: "logstash -f /opt/logstash/server/etc/conf.d/"
  image: kensodev/logstash:latest
  volumes:
    - ./logstash/config:/etc/logstash/conf.d
  ports:
    - "5000:5000"
  links:
    - elasticsearch
kibana:
  build: kibana/
  volumes:
    - ./kibana/config/:/opt/kibana/config/
  ports:
    - "5601:5601"
  links:
    - elasticsearch

This will create all the containers for us, link them and we’ll have a running solution.

In order to check whether your solution is running you can go to your docker-machine ip.

My machine name is elk, I do this:

› docker-machine ip elk
192.168.99.101

If you type that address in a browser, you should see this:

Kibana Blank

As you can see the button is greyed out saying “Unable to fetch mapping”

If you send anything to logstash using:

$ echo Boom! | nc 192.168.99.101 5000

You will see this:

Kibana working

You can now hit “Create” and you have a working ELK solution.

Conclusion

It can be a daunting task to setup an ELK stack, Docker and compose make it easier to run and manage in dev/production.

In the next post, I will go into running this in production.

Thanks for reading and be sure to let me know what you think in the comment section.

Avi Zurel