Comprestimator Utility V1.2.0

Abstract

Comprestimator is a command line host-based utility that can be used to estimate an expected compression rate for block devices.

Download Description

The Comprestimator utility uses advanced mathematical and statistical algorithms to perform the sampling and analysis process in a very short and efficient way. The utility also displays its accuracy level by showing the maximum error range of the results achieved based on the formulas it uses. The utility runs on a host that has access to the devices that will be analyzed, and performs only read operations so it has no effect whatsoever on the data stored on the device. The following section provides useful information on installing Comprestimator on a host and using it to analyze devices on that host. Depending on the environment configuration, in many cases Comprestimator will be used on more than one host, in order to analyze additional data types.

It is important to understand block device behavior when analyzing traditional (fully-allocated) volumes. Traditional volumes that were created without initially zeroing the device may contain traces of old data in the block device level. Such data will not be accessible or viewable in the file system level. When using Comprestimator to analyze such volumes, the expected compression results will reflect the compression rate that will be achieved for all the data in the block device level, including the traces of old data. This simulates the volume mirroring process of the analyzed device into a compressed volume. Later, when volume mirroring is actually used to compress the data on the storage system, it will process all data on the device (including both active data and traces of old data) and get it compressed. After that when storing more active data on the compressed volume, traces of old data will start getting deleted by new data that is written into the volume. As more active data accumulates in the device the compression rate achieved will be adjusted to reflect the accurate savings achieved for the active data. This block device behavior is limited to traditional volumes and will not occur when analyzing thinly provisioned volumes.

Regardless of the type of block device being scanned, it is also important to understand a few characteristics of common file systems space management. When files are deleted from a file system, the space they occupied before being deleted will be freed and available to the file system even though the data on disk was not actually deleted but rather the file system index and pointers were updated to reflect this change. When using Comprestimator to analyze a block device used by a file system – all underlying data in the device will be analyzed, regardless of whether this data belongs to files that were already deleted from the file system. For example – you can fill a 100GB file system and make it 100% used, then delete all the files in the file system making it 0% used. When scanning the block device used for storing the file system in this example, Comprestimator (or any other utility for that matter) will access the data that belongs to the files that were already deleted.

In order to reduce the impact of block device and file system behavior mentioned above it is highly recommended to use Comprestimator to analyze volumes that contain as much active data as possible rather than volumes that are mostly empty of data. This increases accuracy level and reduces the risk of analyzing old data that is already deleted but may still have traces on the device.

Your primary resource for sizing and implementing: 
Real-time Compression in SAN Volume Controller and Storwize V7000 Redpaper –http://www.redbooks.ibm.com/Redbooks.nsf/RedpieceAbstracts/redp4859.html?Open

Instructions on how to set up IBM Real-time Compression for 45 Day Evaluation
Real-time Compression Evaluation User Guide – http://www-304.ibm.com/support/docview.wss?uid=ssg1S7003988&myns=s028&mynp=OCST3FR7

This version of Comprestimator is supported to run on the following host operating systems:

  • Red Hat Enterprise Linux Version 5 (64-bit)
  • ESXi 5.0
  • AIX V6.1, V7.1
  • Windows 2003 Server, Windows 2008 Server (32-bit and 64-bit)

Using ComprestimatorTo use Comprestimator on a Linux, ESXi or AIX server:

1. Log into the server using the root account
2. Run Comprestimator with the -d <device_name> flags to analyze a device or a partition.

To use Comprestimator on a Windows server:

1. Log into the server using the root account
2. Run Comprestimator with the -l flag to list the devices.
3. Run Comprestimator with the-n flag to analyze a device or a partition.

Syntax

In Linux, ESXi and AIX:

comprestimator -d <device> [-p <number_of_processes>] [-P] [-e] [-h] [-c <csv file>] [-v]

In Windows:

comprestimator [-n <disk_number> | -d <device>] [-p <number_of_processes>] [-P] [-e] [-h] [-c <csv file>] [-v]

-d Specifies the device name.
device Path of device to analyze. For example: /dev/sdb, /dev/md-2 (in Linux), /vmware/volumes<datastore_name> (in ESX), /dev/hdisk1 (in AIX).
disk_number Only in Windows – The disk number. Identify this number by running Comprestimator with the –l flag first.
-l List the disk numbers and names available to use by Comprestimator.
-p Specifies the number of processes (or threads in Windows) between 1 and 100. The default value is 10.
-P Displays the results using a paragraph format.
-c Export the results to a CSV-formatted output file.
csv_file File name of CSV-formatted output file.
-e Performs an exhaustive scan. Note that using this option will extend the runtime considerably.
-h Display usage information.
-v Display the results of every sample. Note that the output is extended to a few thousands of lines as a result.

Explanation of scan results

Sample# The number of the current sample reported.
Device The device name used in the scan.
Size(GB) The total size of the device as reported by the operating system, in gigabytes.
Compressed Size(GB) The estimated size of the device if it will be compressed using Storwize V7000/SVC Real-time Compression, in gigabytes.
Total Savings(GB) The total estimated savings from thin-provisioning and compression, in gigabytes.
Total Savings(%) The estimated savings from thin-provisioning and compression, in percentage of the size of the device.
This value is calculated in the following method:
Total Savings(%) = 1-( Compressed Size(GB) / Size(GB) )
Thin Provision Savings(%) The estimated savings from thin provisioning (areas with zeros are stored using very minimal capacity).
Compression Savings(%) The estimated savings from compression.
Compression Accuracy Range(%) The accuracy of the estimate provided by Comprestimator. The results provided are estimated based on samples from the device and therefore may be lower or higher than the actual compression that would be achieved. The approximate accuracy of the results is represented as a percentage of the total size of the device.
For example, if the estimated Compression Savings (%) is 67%, and the Compression Accuracy Range is 5%, the actual compression savings (in percents) if this device will be compressed on Storwize V7000/SVC is between 62% and 72%.

ExamplesTo display the usage information:

./comprestimator

To scan a device in Windows, first identify the disk number using the -l argument:

comprestimator –l

Drive number [0] \\?\ide#diskhitachi_hts725032a9a364_________________pc3zc70f#5&1e544866&0&0.0.0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}

Now run a standard scan of disk 0:

comprestimator –n 0

Version: 1.2.01 (Build w0006)
Start time: 17/07/2012 17:33:08.728421
Device name: \\?\ide#diskhitachi_hts725032a9a364_________________pc3zc70f#5&1e54866&0&0.0.0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}
Device size: 298 GB
Number of processes: 10
Exhaustive: no

To run a standard scan of /dev/sdb device (Linux version):

./comprestimator –d /dev/sdb

Start time: 07/06/2012 12:28:45
Device name: /dev/dm-2
Device size: 500.0 GB
Number of processes: 10
Exhaustive: no

Sample# | Device | size (GB) | Compressed Size(GB) | Total Savings(GB) | Total Savings(%) |Thin Provision Savings(%) | Compression Savings (%) | Compression Accuracy Range(%)
3425 | /dev/dm-2 |500.0|112.8| 387.1 |77.4% |39.6% |62.6% |4.7%

Comprestimator can also be used without any flag to get help and usage information.

How to find the device name in Linux

To list all the available device names in Linux:

fdisk -l

Disk /dev/sda: 78.3 GB, 78383153152 bytes
255 heads, 63 sectors/track, 9529 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000021

Device Boot Start End Blocks Id System
/dev/sda1 * 1 8555 68718006 83 Linux
/dev/sda2 8556 8924 2963992+ 5 Extended
/dev/sda5 8556 8924 2963961 82 Linux swap / Solaris

Use the device names listed in the output, in this example: /dev/sda1, /dev/sda2, /dev/sda5.

How to find the device name in ESXi

The device names that can be scanned using Comprestimator on an ESXi 5 Server can be obtained by using the following command when logged into the ESXi server with root account using SSH:

esxcli storage core device list

To display a shorter, more readable list, you can use the following command to filter out unnecessary information:

esxcli storage core device list | grep Dev

This will list only the of device types and their respective path names. The device path names can then be used as arguments for Comprestimator.

How to find the device name in AIX

To list all the available device names in AIX:

lsdev -Cc disk

hdisk0 Available 00-00-00 SAS Disk Drive
hdisk1 Available 00-00-00 SAS Disk Drive
hdisk2 Available 08-01-02 MPIO FC 2145

To list all the available disk numbers in Windows:

comprestimator –l

Drive number [0] \\?\ide#diskhitachi_hts725032a9a364_________________pc3zc70f#5&1e544866&0&0.0.0#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}

Installation Instructions

To install Comprestimator on a Linux or AIX host:

  1. Log into the host using the root account.
  2. Copy the Comprestimator binary file to any folder on the host.
  3. Verify that the file has execute permissions, to add execute permissions to the binary file, type:
    chmod +x comprestimator

To install Comprestimator on ESXserver:

  1. Enable SSH on the ESXi server. Check the following URL for instructions or skip this step if SSH is already enabled: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017910
  2. Copy the Comprestimator binary file to any folder on the ESXi server. You can copy the file to the ESXi server by using the Secure Copy Protocol (SCP), for example ‘scp’ on Linux or ‘WinSCP’ on Microsoft Windows.

To install Comprestimator on Windows server:

  1. Log into the server.
  2. Copy the appropriate (32-bit or 64-bit) Comprestimator binary file from the corresponding folder to any folder on the host.

Get the Tool here

Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s