I have been working on a project where I need to send logs from Python logger to logstash so I can visualize the logs later using Kibana. Instead of setting up elasticsearch with logstash and kibana on my local workstation, I decided to setup a virtual machine running the complete stack. I have been using Virtualbox on my Ubuntu workstation and pretty happy with it. To make life easier, I put together a vagrant script to spin off VM running the elasticsearch stack so that I can share it work other developers working on the same project.
Below instructions assume that you are running Ubuntu with Virtualbox already installed but I think these instructions can work any other Linux distribution as well.
- Download and install virtualbox. If you are using Ubuntu, you can just use Ubuntu Software Center or apt-get command to install the virtualbox. However, you might have to download and install the virtualbox extension pack manually. After installing the virutalbox, I also created a Host Only network with subnet 172.28.128.0/24.
- Download and install vagrant. You can download the debian package from https://www.vagrantup.com/downloads. Use dpkg or Ubuntu Software Center to install the package.
- Create vgfiles directory in your home directory.
1mkdir ~/vgfiles - Create Vagrant initialization file Vagrantfile under vgfiles directory and add following contents to this file.
1234567891011121314151617181920212223242526# -*- mode: ruby -*-# vi: set ft=ruby :# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!VAGRANTFILE_API_VERSION = "2"Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| # Every Vagrant virtual environment requires a box to build off of.config.vm.box = "ubuntu/trusty64"config.vm.network :forwarded_port, guest: 9200, host: 9200config.vm.network :forwarded_port, guest: 2200, host: 22config.vm.network "private_network", type: "dhcp" # If true, then any SSH connections made will enable agent forwarding.# Default value: false# config.ssh.forward_agent = trueconfig.vm.provider "virtualbox" do |vb|vb.name = "elasticsearch_vm"# Don't boot with headless modevb.gui = true # Tweak the below value to adjust RAMvb.memory = 2048# Tweak the number of processors belowvb.cpus = 2# Use VBoxManage to customize the VM. For example to change memory:# vb.customize ["modifyvm", :id, "--memory", "1024"]endconfig.vm.provision :shell, :path => "bootstrap.sh"end - Create bootstrap.sh file with following contents under vgfiles directory and add following contents to this file.
1234567891011121314151617181920212223242526272829#!/usr/bin/env bashwget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | apt-key add -echo 'deb http://packages.elasticsearch.org/elasticsearch/1.3/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.listecho 'deb http://packages.elasticsearch.org/logstash/1.4/debian stable main' | sudo tee /etc/apt/sources.list.d/logstash.listecho 'deb http://packages.elasticsearch.org/logstashforwarder/debian stable main' | sudo tee /etc/apt/sources.list.d/logstashforwarder.list# update aptsudo apt-get update# install javasudo apt-get install openjdk-7-jre-headless nginx apache2-utils -ysudo apt-get install elasticsearch logstash logstash-forwarder -y# Configure logstash forwardercd /etc/init.d/sudo wget https://raw.github.com/elasticsearch/logstash-forwarder/master/logstash-forwarder.init -O logstash-forwardersudo chmod +x logstash-forwardersudo update-rc.d logstash-forwarder defaultssudo service elasticsearch start# install headsudo /usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head# Install Kibanacd ~wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gztar zxvf kibana-3.1.0.tar.gzmv kibana-3.1.0 /usr/share/kibana3sed -i 's/hostname+":9200/hostname+":80/' /usr/share/kibana3/config.jswget https://github.com/elasticsearch/kibana/raw/kibana3/sample/nginx.confcp nginx.conf /etc/nginx/sites-available/defaultsudo service nginx restart - Change your current directory to vgfiles.
1#cd ~/vgfiles - Run vagrant command to build the virtual machine.
1#vagrant up
Based on your Internet connection speed, it came take few minutes to build the virtual machine. - Once the virtual machine build process is completed, ssh to this machine using port 2222.
1ssh -p 2222 vagrant@localhost
Use password vagrant when prompted. - Use ifconfig command to find the second NIC ip address.
- You can use following URLs to use elasticsearch
Elasticsearch – http://localhost:9200/_plugin/head/
Kibana – http://<IP address>/
Your examples seem to have lost something in translation. I think there are a few newlines that were eaten, so when I copy the code into a vagrant file, I get some errors. An example:
# config.ssh.forward_agent = true config.vm.provider “virtualbox” do |vb|
All appears on a single line, when it should be on two lines like:
# config.ssh.forward_agent = true
config.vm.provider “virtualbox” do |vb|
Thanks for the post though, it’s a good start!
Gabe.. thanks for the note. I just fixed the issue.
Thanks a lot,
it helped me very much.
I tried that with stable’s kibana3.1.2 and es 1.4.2 and it works !!!
note – the link to sample/nginx.conf has changed because the branch master now is kibana 4. You’ve to download it from https://github.com/elasticsearch/kibana/tree/kibana3/sample
Best regards.
Or better change master by kibana3
wget https://github.com/elasticsearch/kibana/raw/kibana3/sample/nginx.conf
Bye
You code blocks are difficult to cut and paste. You get one long line 🙁