If you are looking for an easy, reliable and free server backup solution for LAMP server, database servers or any other Linux server, you came to the right place.
The server backup solution I have been using for few years now is based on duplicity project. Below diagram shows my current backup schema to ensure that I have a copy of backup at both onsite and offsite locations.
Few points to notice:
- Linux workstation is the computer for which will be backed up.
- Local NFS server is used to store onsite copy of server backups.
- A copy of backup is also stored on Amazon S3 cloud for disaster recovery. Duplicity supports backing up to Amazon S3, Azure, Dropbox, Google Docs, Google Cloud Storage, OneDrive and few others directly by utilizing s3cmd. But I prefer to use Duplicity to backup at onsite location and then utilizing s3cmd to sync the onsite backup to cloud storage.
NOTE: This tutorial is was created for Ubuntu platform (14.04 to be specific) but it can be easily adopted for any other Linux distribution.
Let’s get started with real stuff! Continue reading Linux Server Backup using Free & Open Source Tools
I have been working on a project where I need to send logs from Python logger to logstash so I can visualize the logs later using Kibana. Instead of setting up elasticsearch with logstash and kibana on my local workstation, I decided to setup a virtual machine running the complete stack. I have been using Virtualbox on my Ubuntu workstation and pretty happy with it. To make life easier, I put together a vagrant script to spin off VM running the elasticsearch stack so that I can share it work other developers working on the same project.
Continue reading Elasticsearch with Logstash & Kibana using Vagrant & Virtualbox on Ubuntu 14.04
If you work in IT industry, you must have came across the question of how well your network storage or even your local storage is performing. There can be many reasons for looking into storage performance including but not limited to..
- You are evaluating new storage solution to deploy in your organization.
- You are trying to find the bottlenecks in your existing storage solution.
- You are product engineer and working on performance testing of storage product.
Storage Performance Testing
Measuring storage performance of a storage can be easy but finding bottleneck in storage solution can be a cumbersome task as there can be lots of hardware/software layers in a given storage solution. Let’s take an example of NAS storage product. Below diagram shows what a typical NAS storage product architecture looks like. When it comes to storage performance testing, you might be testing complete solution or any layer/component as shown in above diagram. Regardless of the component you are testing, storage performance is measured using following key metrics. Before we start digging into the details, please read the article on to better under I/O size that can impact the performance results.
Continue reading Understanding storage performance testing & metrics
I/O request size which is also referred as block size is the size of request that application uses to perform I/O operation on storage device. Every application uses a different I/O request size and based on how big the request size is, it can have pretty good impact on storage performance.
Storage performance is measure in IOPS and/or Throughput. The size of I/O request impact both of these number. Below formula shows the relationship of these three
Throughput = IOPS x I/O size
Higher the I/O size, bigger the throughput will be. On the revere size, IOPS will be lower whereas if the I/O size is smaller you will get higher IOPS. Remember, the data is stored on the storage device in size of 512 Bytes (Most common). It take a certain amount of time for storage device to read/write single sector (512 Bytes for hard disk drives). I suggest that you read http://en.wikipedia.org/wiki/Advanced_Format to learn more about sector size.
Now let’s say that you have read I/O request (operations) which is 8KB in size. To complete this request, disk will have to read 16 sectors. As you can guess, it will take more time to read 16 sectors as compared to 1 sector from the disk drive. If we change the I/O request size to 64KB, 128 sectors will be read to complete the request and it will take even more time as compared to complete 8KB size request.
Below chart shows what the typical I/O size looks like along with the access pattern for a given application type.
||Blocks (In KB)
|Database (transaction processing)
|File Copy (SMB)
|Database (log ﬁle)
To find the correct block size for your application, you can use your operating systems monitoring capabilities. Windows 2012 has built-in Performance monitor that can provide this data. Look for Avg. Disk Bytes/Read counter.
If you are using Linux, you can do the following to find the I/O request size
- Run iostat and look at the avgrq-sz data. This column shows the average number of sectors requested in a given I/O request
- Multiply this number by 512 bytes to get the block I/O request size in bytes
I was trying to use Duplicator plugin on one of the website using WordPress which is hosted on Dreamhost. Duplicator plugin make use of PHP zip module to compress the wordpress files and build a single package for installation on other location or backup/restore. To my surprise, Dreamhost doesn’t have PHP zip module support but what I love about dreamhost is their good documentation about these type of issues. They do have Wiki article on installing zip module http://wiki.dreamhost.com/PECL_Zip.
I create a zip.sh shell script with the content from Wiki article in Dreamhost user home directory. When I executed the script, I was returned with following error message.
<span style="color: #000000;">/home/userdir/php5/bin/phpize: No such file or directory</span>
The phpize command is used to prepare the build environment for a PHP extension. The workaround to move forward is pretty easy. Use following commands:
ln -s /usr/local/bin/phpize-5.4 php5/bin/phpize
Run the zip.sh again and everything works as expected.
Number of times I have been asked this question as why my Operation System shows less disk space than what I have? Few years back when the disk sizes were limited to only few hundred gigabytes, this wasn’t as big of concern as the lost disk space was few gigabytes. But nowadays with disk size all the way up to 2 TB and 3TB on horizon, the lost disk space can be few hundred Gigabytes. And to make it worst, if you are building disk array (RAID or JBOD), this loss could be even more than whole 1TB.
There are few explanations for lost disk space and are explained below: Continue reading Why operating system shows less disk space than what I have?
Just upgraded my laptop to Windows 7. After installing Windows 7 and configuring devices, I started downloading all the good application without which I can’t survive a day on computer. Once of these applications was Notepad++. After installing Notepad++, I realized that the Notepad++ edit option was missing from context menu.
I am use to just right click on files and select Notepad++ so this was bit of disappointment. After doing some digging on google, I was able to create a registry entry to context menu and here are the steps which I had to follow:
- Launch registry editor using regedit command.
- Browse to HKEY_CLASSES_ROOT\*\shell registry tree.
- Create a new key under band name it Notepad++
- Under this key, create a new registry entry of type REG_SZ and enter Notepad++ as data for this registry key. (The data entry will appear as it is in context menu. You can also enter ‘Open with Notepad++’ if that’s what you want to appear in context menu.)
- Create another key entry under Notepad++ and name it command.
- Create a new registry entry of type REG_SZ and add C:\Program Files (x86)\Notepad++\Notepad++.exe %1 as data. If you installed Notepad++ at different location that default, modify the path in command accordingly.
- That’s it, you can close the registry editor and you should have Notepad++ in context menu for all file types. Below screenshot shows the registry editor screen with entry.
If it seems hard to follow all the above steps, you can just download the registry file and execute it
I just finished setting up subversion server running on CentOS. In addition to having subversion respositories, I have few other requirements which includes:
- Ability to browse repositories using a web browser
- Acccess to repositories using WebDAV
- Repositories access should be protected and authentication/authorization should be done using Active Directory
First, I was looking at using TRAC which provides most of the above and some extra features (project management and bug/issue tracking). But we already have other project management (dotProject) and bug/issue tracking (Mantis) softwares in place and TRAC doesn’t seems like very straight forward to install and configure. So, I started looking at alternatives and there isn’t much out there in form of once package which can provide all of the above.
The solution which we ended up building is running on CentOS operating system, WebSVN and other dependency libraries. I ran into some issues while setting up the solution and went through number of articles and KBs available on Internet. Almost spent whole day on this and thought it might be useful for others if I put all the info in one article.
Target is to have a setup where we can browse all the repositories using WebSVN with URL
Each of the repository should be accessible using a URL which looks like
http://<hostname>/<repository name> Continue reading Subversion with WebSVN, WebDAV and Active Directory or LDAP Authentication
With release of Windows Server 2008, Micrsoft Windows team made some major improvements to overall architecture to make Windows operating system more stable, secure and reliable. Keeping the Windows 2003 tradition alive, Microsoft ships Windows 2008 in four versions – Web Server, Standard, Enterprise and Datacenter. But one major feature addition to 2008 is the ability to do the minimal installation of Windows 2008 without any graphical utilities which most of the admins don’t need. This feature is known as “Server Core” and its basically provides a minimal environment for running a specific server role. Admins can select to perform Server Core installation during the Windows 2008 installation. The result will be a stripped down installation of Windows without any traditional components installed such as desktop shell, GUI, Windows Explorer, IE and other MMC which otherwise get installed while doing full installation. With Server Core installation, all the management and configuration tasks will be performed using command line utilities.
You must be thinking, why would someone use the Server Core installation? Everything to be configured from command line…. that sounds like going back to era of DOS? Well, there are multiple advantages when using Server Core:
- Reduced maintenance
- Reduced attack surface
- Reduced management
- Less disk space required for installation as footprint is very small