Linux Server Backup using Free & Open Source Tools

If you are looking for an easy, reliable and free server backup solution for LAMP server, database servers or any other Linux server, you came to the right place.

The server backup solution I have been using for few years now is based on duplicity project. Below diagram shows my current backup schema to ensure that I have a copy of backup at both onsite and offsite locations.

LAMP Server backup to cloud storage

Few points to notice:

  1. Linux workstation is the computer for which will be backed up.
  2. Local NFS server is used to store onsite copy of server backups.
  3. A copy of backup is also stored on Amazon S3 cloud for disaster recovery. Duplicity supports backing up to Amazon S3, Azure, Dropbox, Google Docs, Google Cloud Storage, OneDrive and few others directly by utilizing s3cmd. But I prefer to use Duplicity to backup at onsite location and then utilizing s3cmd to sync the onsite backup to cloud storage.

NOTE: This tutorial is was created for Ubuntu platform (14.04 to be specific) but it can be easily adopted for any other Linux distribution.

Let’s get started with real stuff! Continue reading Linux Server Backup using Free & Open Source Tools

Elasticsearch with Logstash & Kibana using Vagrant & Virtualbox on Ubuntu 14.04

I have been working on a project where I need to send logs from Python logger to logstash so I can visualize the logs later using Kibana. Instead of setting up elasticsearch with logstash and kibana on my local workstation, I decided to setup a virtual machine running the complete stack. I have been using Virtualbox on my Ubuntu workstation and pretty happy with it. To make life easier, I put together a vagrant script to spin off VM running the elasticsearch stack so that I can share it work other developers working on the same project.

Continue reading Elasticsearch with Logstash & Kibana using Vagrant & Virtualbox on Ubuntu 14.04

Understanding storage performance testing & metrics

If you work in IT industry, you must have came across the question of how well your network storage or even your local storage is performing. There can be many reasons for looking into storage performance including but not limited to..

  • You are evaluating new storage solution to deploy in your organization.
  • You are trying to find the bottlenecks in your existing storage solution.
  • You are product engineer and working on performance testing of storage product.

Storage Performance Testing

Measuring storage performance of a storage can be easy but finding bottleneck in storage solution can be a cumbersome task as there can be lots of hardware/software layers in a given storage solution. Let’s take an example of NAS storage product. Below diagram shows what a typical NAS storage product architecture looks like. NAS Architecture Diagram When it comes to storage performance testing, you might be testing complete solution or any layer/component as shown in above diagram. Regardless of the component you are testing, storage performance is measured using following key metrics. Before we start digging into the details, please read the article on to better under I/O size that can impact the performance results.

Continue reading Understanding storage performance testing & metrics

Understanding I/O request or block size

I/O request size which is also referred as block size is the size of request that application uses to perform I/O operation on storage device. Every application uses a different I/O request size and based on how big the request size is, it can have pretty good impact on storage performance.

Storage performance is measure in IOPS and/or Throughput. The size of I/O request impact both of these number. Below formula shows the relationship of these three

Throughput   =   IOPS   x   I/O size

Higher the I/O size, bigger the throughput will be. On the revere size, IOPS will be lower whereas if the I/O size is smaller you will get higher IOPS. Remember, the data is stored on the storage device in size of 512 Bytes (Most common). It take a certain amount of time for storage device to read/write single sector (512 Bytes for hard disk drives). I suggest that you read http://en.wikipedia.org/wiki/Advanced_Format to learn more about sector size.

Now let’s say that you have read I/O request (operations) which is 8KB in size. To complete this request, disk will have to read 16 sectors. As you can guess, it will take more time to read 16 sectors as compared to 1 sector from the disk drive. If we change the I/O request size to 64KB, 128 sectors will be read to complete the request and it will take even more time as compared to complete 8KB size request.

Below chart shows what the typical I/O size looks like along with the access pattern for a given application type.

Application Type Access Pattern Read % Write % Blocks (In KB)
Database (transaction processing) Random 70 30 8
Mail Server Random 70 30 8
File Copy (SMB) Random 50 50 64
Database (log file) Sequential 0 100 64
Web Server Random 0 100 64
Backup Sequential 0 100 64
Restore Sequential 100 0 64

To find the correct block size for your application, you can use your operating systems monitoring capabilities. Windows 2012 has built-in Performance monitor that can provide this data. Look for Avg. Disk Bytes/Read counter.

If you are using Linux, you can do the following to find the I/O request size

  • Run iostat and look at the avgrq-sz data. This column shows the average number of sectors requested in a given I/O request
  • Multiply this number by 512 bytes to get the block I/O request size in bytes

Dreamhost PHP zip extension installation

I was trying to use Duplicator plugin on one of the website using WordPress which is hosted on Dreamhost. Duplicator plugin make use of PHP zip module to compress the wordpress files and build a single package for installation on other location or backup/restore. To my surprise, Dreamhost doesn’t have PHP zip module support but what I love about dreamhost is their good documentation about these type of issues. They do have Wiki article on installing zip module http://wiki.dreamhost.com/PECL_Zip.

I create a zip.sh shell script with the content from Wiki article in Dreamhost user home directory. When I executed the script, I was returned with following error message.

The phpize command is used to prepare the build environment for a PHP extension. The workaround to move forward is pretty easy. Use following commands:

Run the zip.sh again and everything works as expected.

Why operating system shows less disk space than what I have?

Number of times I have been asked this question as why my Operation System shows less disk space than what I have? Few years back when the disk sizes were limited to only few hundred gigabytes, this wasn’t as big of concern as the lost disk space was few gigabytes. But nowadays with disk size all the way up to 2 TB and 3TB on horizon, the lost disk space can be few hundred Gigabytes. And to make it worst, if you are building disk array (RAID or JBOD), this loss could be even more than whole 1TB.

There are few explanations for lost disk space and are explained below: Continue reading Why operating system shows less disk space than what I have?

Notepad++ missing from Context Menu in Windows 7

Just upgraded my laptop to Windows 7. After installing Windows 7 and configuring devices, I started downloading all the good application without which I can’t survive a day on computer. Once of these applications was Notepad++. After installing Notepad++, I realized that the Notepad++ edit option was missing from context menu.

I am use to just right click on files and select Notepad++ so this was bit of disappointment. After doing some digging on google, I was able to create a registry entry to context menu and here are the steps which I had to follow:

  1. Launch registry editor using regedit command.
  2. Browse to HKEY_CLASSES_ROOT\*\shell registry tree.
  3. Create a new key under band name it Notepad++
  4. Under this key, create a new registry entry of type REG_SZ and enter Notepad++ as data for this registry key. (The data entry will appear as it is in context menu. You can also enter ‘Open with Notepad++’ if that’s what you want to appear in context menu.)
  5. Create another key entry under Notepad++ and name it command.
  6. Create a new registry entry of type REG_SZ and add C:\Program Files (x86)\Notepad++\Notepad++.exe %1 as data. If you installed Notepad++ at different location that default, modify the path in command accordingly.
  7. That’s it, you can close the registry editor and you should have Notepad++ in context menu for all file types. Below screenshot shows the registry editor screen with entry.
    Notepad++ Context Entry

If it seems hard to follow all the above steps, you can just download the registry file and execute it 🙂

Subversion with WebSVN, WebDAV and Active Directory or LDAP Authentication

I just finished setting up subversion server running on CentOS. In addition to having subversion respositories, I have few other requirements which includes:

  • Ability to browse repositories using a web browser
  • Acccess to repositories using WebDAV
  • Repositories access should be protected and authentication/authorization should be done using Active Directory

First, I was looking at using TRAC which provides most of the above and some extra features (project management and bug/issue tracking). But we already have other project management (dotProject) and bug/issue tracking (Mantis) softwares in place and TRAC doesn’t seems like very straight forward to install and configure. So, I started looking at alternatives and there isn’t much out there in form of once package which can provide all of the above.

The solution which we ended up building is running on CentOS operating system, WebSVN and other dependency libraries. I ran into some issues while setting up the solution and went through number of articles and KBs available on Internet. Almost spent whole day on this and thought it might be useful for others if I put all the info in one article.

Target is to have a setup where we can browse all the repositories using WebSVN with URL
http://<hostname>/websvn
Each of the repository should be accessible using a URL which looks like
http://<hostname>/<repository name> Continue reading Subversion with WebSVN, WebDAV and Active Directory or LDAP Authentication

Windows Server 2008 Server Core Installation and Management

With release of Windows Server 2008, Micrsoft Windows team made some major improvements to overall architecture to make Windows operating system more stable, secure and reliable. Keeping the Windows 2003 tradition alive, Microsoft ships Windows 2008 in four versions – Web Server, Standard, Enterprise and Datacenter. But one major feature addition to 2008 is the ability to do the minimal installation of Windows 2008 without any graphical utilities which most of the admins don’t need. This feature is known as “Server Core” and its basically provides a minimal environment for running a specific server role. Admins can select to perform Server Core installation during the Windows 2008 installation. The result will be a stripped down installation of Windows without any traditional components installed such as desktop shell, GUI, Windows Explorer, IE and other MMC which otherwise get installed while doing full installation. With Server Core installation, all the management and configuration tasks will be performed using command line utilities.

You must be thinking, why would someone use the Server Core installation? Everything to be configured from command line…. that sounds like going back to era of DOS? Well, there are multiple advantages when using Server Core:

  • Reduced maintenance
  • Reduced attack surface
  • Reduced management
  • Less disk space required for installation as footprint is very small
But don’t get disappointed, Microsoft does allow you to manage Server Core installation using your favorite MMC from remote server or workstation. Continue reading Windows Server 2008 Server Core Installation and Management

HTML and XHTML Quick Reference Sheet

Common Elements

Element Value Examples
float: <position> float is used to float an element on the left or right margin.<position> could be none, left or right. <img src=”sample.jpg” style=”float: right” />
clear: <position> clear is used to display an element in the first available space where the specified margin is clear of floating elements.
<position> could be left, right, all or none.		 <hr style=”clear: right” />
margin-top: <length>
margin-right: <length>
margin-bottom: <length>
margin-left: <length>		 These elements are used to set the size of margins around an element.
<length> can be unit of length, a percentage of the width of the containing element, or the keyword “auto” (the default) which enables the browser to set the margin size.		 <table style=”margin-top: 15px”>
<body style=”margin-left: 20px”>
margin: <top> <right> <bottom> <left> margin element combine all margin styles in a single style. <top>, <right>, <bottom> and <left> are the margins of the top, right, bottom and left edges. If you include only three values, the margins are applied to the top, right and bottom. If only two values are specified, the first value is applied to top and bottom edges whereas second value is applied to right and left edges. If you apply only one value, it is applied to all four edges. <img src=”sample.jpg” style=”margin: 0 0 5 15″>
<body style=”margin: 10″>
background-image: url(<url>) background-image attribute is used to set the background image for any element. <url> is the name and location of the image file to set as background. <body style=”background-image: url(sample.jpg)”>
background-repeat: <type> This attribute is used used in combination with background-image attribute. It controls how browser tile an image across and element’s background. The <type> can be one of keywords out of repeat (the default), repeat-x, repeat-y, or no-repeat. <body style=”background-image: url(sample.jpg); background-repeat: repeat-x”>
background-position: <horizontal> <vertical> This attribute is used to place the background image in a specific position behind the element. <horizontal> is the horizontal position of the image, and <vertical> is the vertical position. You can specify a position as the distance from the top-left corner of the element, a percentage of the element’s width and height, or by using a keyword. Keyword options are top, center, or bottom for vertical position, and left, center, or right for horizontal placement. <body style=”background-image: url(sample.jpg); background-position: 10% 20%”>
<body style=”background-image: url(sample.jpg); background-position: right bottom”>

Continue reading HTML and XHTML Quick Reference Sheet