Category: Articles

Linux Server Backup using Free & Open Source Tools

If you are looking for an easy, reliable and free server backup solution for LAMP server, database servers or any other Linux server, you came to the right place.

The server backup solution I have been using for few years now is based on duplicity project. Below diagram shows my current backup schema to ensure that I have a copy of backup at both onsite and offsite locations.

LAMP Server backup to cloud storage

Few points to notice:

  1. Linux workstation is the computer for which will be backed up.
  2. Local NFS server is used to store onsite copy of server backups.
  3. A copy of backup is also stored on Amazon S3 cloud for disaster recovery. Duplicity supports backing up to Amazon S3, Azure, Dropbox, Google Docs, Google Cloud Storage, OneDrive and few others directly by utilizing s3cmd. But I prefer to use Duplicity to backup at onsite location and then utilizing s3cmd to sync the onsite backup to cloud storage.

NOTE: This tutorial is was created for Ubuntu platform (14.04 to be specific) but it can be easily adopted for any other Linux distribution.

Let’s get started with real stuff! Continue reading Linux Server Backup using Free & Open Source Tools

Understanding storage performance testing & metrics

If you work in IT industry, you must have came across the question of how well your network storage or even your local storage is performing. There can be many reasons for looking into storage performance including but not limited to..

  • You are evaluating new storage solution to deploy in your organization.
  • You are trying to find the bottlenecks in your existing storage solution.
  • You are product engineer and working on performance testing of storage product.

Storage Performance Testing

Measuring storage performance of a storage can be easy but finding bottleneck in storage solution can be a cumbersome task as there can be lots of hardware/software layers in a given storage solution. Let’s take an example of NAS storage product. Below diagram shows what a typical NAS storage product architecture looks like. NAS Architecture Diagram When it comes to storage performance testing, you might be testing complete solution or any layer/component as shown in above diagram. Regardless of the component you are testing, storage performance is measured using following key metrics. Before we start digging into the details, please read the article on to better under I/O size that can impact the performance results.

Continue reading Understanding storage performance testing & metrics

Understanding I/O request or block size

I/O request size which is also referred as block size is the size of request that application uses to perform I/O operation on storage device. Every application uses a different I/O request size and based on how big the request size is, it can have pretty good impact on storage performance.

Storage performance is measure in IOPS and/or Throughput. The size of I/O request impact both of these number. Below formula shows the relationship of these three

Throughput   =   IOPS   x   I/O size

Higher the I/O size, bigger the throughput will be. On the revere size, IOPS will be lower whereas if the I/O size is smaller you will get higher IOPS. Remember, the data is stored on the storage device in size of 512 Bytes (Most common). It take a certain amount of time for storage device to read/write single sector (512 Bytes for hard disk drives). I suggest that you read http://en.wikipedia.org/wiki/Advanced_Format to learn more about sector size.

Now let’s say that you have read I/O request (operations) which is 8KB in size. To complete this request, disk will have to read 16 sectors. As you can guess, it will take more time to read 16 sectors as compared to 1 sector from the disk drive. If we change the I/O request size to 64KB, 128 sectors will be read to complete the request and it will take even more time as compared to complete 8KB size request.

Below chart shows what the typical I/O size looks like along with the access pattern for a given application type.

Application Type Access Pattern Read % Write % Blocks (In KB)
Database (transaction processing) Random 70 30 8
Mail Server Random 70 30 8
File Copy (SMB) Random 50 50 64
Database (log file) Sequential 0 100 64
Web Server Random 0 100 64
Backup Sequential 0 100 64
Restore Sequential 100 0 64

To find the correct block size for your application, you can use your operating systems monitoring capabilities. Windows 2012 has built-in Performance monitor that can provide this data. Look for Avg. Disk Bytes/Read counter.

If you are using Linux, you can do the following to find the I/O request size

  • Run iostat and look at the avgrq-sz data. This column shows the average number of sectors requested in a given I/O request
  • Multiply this number by 512 bytes to get the block I/O request size in bytes

Dreamhost PHP zip extension installation

I was trying to use Duplicator plugin on one of the website using WordPress which is hosted on Dreamhost. Duplicator plugin make use of PHP zip module to compress the wordpress files and build a single package for installation on other location or backup/restore. To my surprise, Dreamhost doesn’t have PHP zip module support but what I love about dreamhost is their good documentation about these type of issues. They do have Wiki article on installing zip module http://wiki.dreamhost.com/PECL_Zip.

I create a zip.sh shell script with the content from Wiki article in Dreamhost user home directory. When I executed the script, I was returned with following error message.

The phpize command is used to prepare the build environment for a PHP extension. The workaround to move forward is pretty easy. Use following commands:

Run the zip.sh again and everything works as expected.

Why operating system shows less disk space than what I have?

Number of times I have been asked this question as why my Operation System shows less disk space than what I have? Few years back when the disk sizes were limited to only few hundred gigabytes, this wasn’t as big of concern as the lost disk space was few gigabytes. But nowadays with disk size all the way up to 2 TB and 3TB on horizon, the lost disk space can be few hundred Gigabytes. And to make it worst, if you are building disk array (RAID or JBOD), this loss could be even more than whole 1TB.

There are few explanations for lost disk space and are explained below: Continue reading Why operating system shows less disk space than what I have?

Notepad++ missing from Context Menu in Windows 7

Just upgraded my laptop to Windows 7. After installing Windows 7 and configuring devices, I started downloading all the good application without which I can’t survive a day on computer. Once of these applications was Notepad++. After installing Notepad++, I realized that the Notepad++ edit option was missing from context menu.

I am use to just right click on files and select Notepad++ so this was bit of disappointment. After doing some digging on google, I was able to create a registry entry to context menu and here are the steps which I had to follow:

  1. Launch registry editor using regedit command.
  2. Browse to HKEY_CLASSES_ROOT\*\shell registry tree.
  3. Create a new key under band name it Notepad++
  4. Under this key, create a new registry entry of type REG_SZ and enter Notepad++ as data for this registry key. (The data entry will appear as it is in context menu. You can also enter ‘Open with Notepad++’ if that’s what you want to appear in context menu.)
  5. Create another key entry under Notepad++ and name it command.
  6. Create a new registry entry of type REG_SZ and add C:\Program Files (x86)\Notepad++\Notepad++.exe %1 as data. If you installed Notepad++ at different location that default, modify the path in command accordingly.
  7. That’s it, you can close the registry editor and you should have Notepad++ in context menu for all file types. Below screenshot shows the registry editor screen with entry.
    Notepad++ Context Entry

If it seems hard to follow all the above steps, you can just download the registry file and execute it 🙂

Subversion with WebSVN, WebDAV and Active Directory or LDAP Authentication

I just finished setting up subversion server running on CentOS. In addition to having subversion respositories, I have few other requirements which includes:

  • Ability to browse repositories using a web browser
  • Acccess to repositories using WebDAV
  • Repositories access should be protected and authentication/authorization should be done using Active Directory

First, I was looking at using TRAC which provides most of the above and some extra features (project management and bug/issue tracking). But we already have other project management (dotProject) and bug/issue tracking (Mantis) softwares in place and TRAC doesn’t seems like very straight forward to install and configure. So, I started looking at alternatives and there isn’t much out there in form of once package which can provide all of the above.

The solution which we ended up building is running on CentOS operating system, WebSVN and other dependency libraries. I ran into some issues while setting up the solution and went through number of articles and KBs available on Internet. Almost spent whole day on this and thought it might be useful for others if I put all the info in one article.

Target is to have a setup where we can browse all the repositories using WebSVN with URL
http://<hostname>/websvn
Each of the repository should be accessible using a URL which looks like
http://<hostname>/<repository name> Continue reading Subversion with WebSVN, WebDAV and Active Directory or LDAP Authentication

Most Useful Free Applications for Windows

Unlike any Linux distrubution, Microsoft Windows lacks number of utilities or tools that you will need from time to time. If you are IT professional, there are number of utilities you will need to get through your day in addition to what Microsoft Windows comes with. Thanks to open source methodology and licensing, number of these utilities are available free along with their source codes. And there are bunch of softwares available for Windows which are free but closed source which is totally fine as long as you don’t have to go through number of hoops to get your hands on the software.

For me, the free software is one:

  • which is totally free as in free beer 😉
  • which doesn’t comes with adwares, spywares or other kind of hidden methods to make some profits from the users.
  • which doesn’t require activation or registration after few days or otherwise some of the features will be locked down and so called ‘free’ app will be stripped down version of what you were expecting.

On the other hand, I love the free applications which provide in-built update checks and auto-update features and…. well lets not get too greedy 🙂

Lets get started with our list of applications. The list below is not sorted out by popularity or functionality because why take extra headache of sorting and I will keep adding new apps in the future.

Continue reading Most Useful Free Applications for Windows