As you may remember when SATA drive technology came around several years ago, it was a very exciting time. This new low cost, high-capacity, commodity disk drive revolutionized the home computer data storage needs.

This fueled the age of the digital explosion. Digital photos and media quickly filled hard drives around the world and affordably. This digital explosion propelled companies like Apple and Google into the hundreds of billions in revenue. This also propelled the explosive data growth in the enterprise.

The SAN industry scrambled to meet this demand. SAN vendors such as EMC, NetApp and others saw the opportunity to move into a new market using these same affordable high-capacity drives to quench the thirst for storage.

The concept of using SATA drives in a SAN went mainstream. Companies that once could not afford a SAN can now buy a SAN with larger capacities for a fraction of the cost of a traditional SAN. This was so popular that companies bought SATA based SANs by the bulk, often in multiple batches at a time.

As time progressed, these drives started failing. SATA was known for their low MTBF (mean time before failure) rates. SATA SANs employed RAID 5 at first to provide protection for a single drive failure, but not for dual drive failure.

As companies started to employ RAID 6 technology dual drive failure protection would not result in data loss.

The “Perfect Storm” even with RAID 6 protection looks like this…

– Higher Capacity Drives = longer rebuild times: The industry has released 3TB drives. Depending on SAN vendor, this will vary. I have seen 6 days for a rebuild of a 2TB drive

– Denser Array Footprint = increased heat and vibrations: Dramatically reducing MTBF

– Outsourced drive manufacturing to third world countries = increase rate in drive failures particularly in batches or series: Quality control and management is lacking in outsourced facilities resulting in mass defects

– Common MTBF in Mass Numbers = drives will fail around the same time: This is a statistical game. For example, a 3% failure rate for a SAN in a datacenter is acceptable, but when there are mass quantities of these drives, 3% will approach and/or exceed the fault tolerant of RAID

Virtualized Storage = Complexity in recovery : Most SAN vendors now have virtualized storage, but recovery will vary depending on how they do their virtualization

– Media Errors on Drives = Failure to successfully rebuild RAID volumes: The larger the drive the chance of media errors become greater. Media errors are errors that are on the drive that renders small bits of data to be unreadable. Rebuild of RAID volumes may be compromised or failed due to these errors.

Don’t be fooled into having a false sense of security but having just RAID 6. Employ good backups and data replication as an extension of a good business continuity or disaster recovery plan.

As the industry moves to different technologies other new and interesting anomalies will develop.

In technology, life is never a dull moment.

Other than protecting your data, is there a difference in RAID levels? Definitely! Storing data is much like driving. Depending on what you are doing with your car or what the road conditions are, there are optimal vehicles for those situations. For example, if you have a family of 5, it would be ideal to drive a minivan, although you can get around in a Toyota Camry. If you are driving in the snow, it would be ideal to drive a jeep, although you can also use a BMW with traction control. Knowing the properties of RAID will help you architect the best storage system for your applications. In the world of RAID, there is something known as write penalty.

Here is the breakdown:

RAID 1 or 10 = Write Penalty of 2
RAID 5 or 50 = Write Penalty of 4
RAID 6 = Write Penalty of 6

To put these metrics in perspective;

RAID 1 or 10 is like a BMW M5, fast and expensive. It is the best for applications that demand high performance in both Reads and Writes. The drawback is that it takes 100% capacity overhead. So if you have 100GB of RAID 1 or 10 storage it will consume another 100GB of additional capacity. RAID 1 or 10 is ideal for Databases, Virtual Desktops or any application that demands high IOPS, particularly Write-intensive apps.

RAID 5 or 50 is like a Nissan Altima, multi-purpose and affordable. It provides for the most usable disk capacity while providing robust disk redundancy. The capacity overhead is around 30%, providing exceptional value. RAID 5 or 50 is the most widely used in IT environments today. In many cases, RAID 5 is used exclusively. In fact, most environments, with Database, Email and File servers use RAID 5 without issues. Just like the old saying, “the squeaky wheel gets the grease”, if the users don’t complain, IT departments won’t bother changing RAID levels.

RAID 6 is like the Minivan. Made for capacity and not for speed. It is typically used for 1TB drives and above. RAID 6 is designed to withstand double disk failures. When a 1TB drive (or larger) fails, it takes a lot longer for the replacement drive to rebuild (I have seen around 15 – 24+ hours). Generally, the longer the rebuilding takes, the larger the window for another drive to fail, hence, exposure to data loss. So RAID 6 is good for this purpose as it provides ample protection for drives (usually 1TB or higher) that have lower MTBF (Mean Time Before Failure). The drawback with RAID 6 is the write penalty is high. This is not a good RAID level to use for Write-intensive applications. RAID 6 is usually used for data archiving, data warehousing and storing static data (i.e., PDF, JPG, etc).

Although there are considerable write penalties for RAID 5 or 50 and 6, the Read penalty is the same as RAID 1 or 10. The benefit of a Read intensive environment is that RAID 5 or 50 and 6 can be used without the high cost of RAID 10. So before you choose your RAID level, take a look at what you are doing with the data that will reside on it. You will thank me later for that…

Do you know we live in amazing times? When I was growing up and if I wanted to learn to dance, I would have to take lessons. If I wanted to learn construction, I would get an entry level job as a construction worker. Today, you can Google virtually anything and learn almost anything from the Internet. As I was thinking today about how far we have come with storing data. I wanted to take this time to simplify this nifty technology so that many can “Google” RAID technology and understand it in about 5 minutes.

Although RAID has been around for a long time, most people who are not in IT won’t know what RAID is. There has been some consumer version of hardware RAID cards for the home, but is not commonly used. Let’s first start with what RAID stands for; “Redundant Array of Independent Disks”. When RAID was first introduced, it stood for “Redundant Array of Inexpensive Disks”. The acronym was changed to reflect the changing nature of hard drives and RAID sets. Basically, RAID is a data protection method that employs different data storing algorithms using a set of disks. There are different levels of RAID which is designated by a number following the term “RAID X” (X being the RAID level). I will break down the different RAID levels for you.

RAID 0
This RAID level is striping without parity. Striping is the ability to store data across multiple drives. Parity is an error correction method that is used in RAID and is a core mechanism in rebuilding failed drives. This RAID level offers no protection… Yes, this is the only RAID level you probably don’t want to use. The only advantage of this RAID level is increased capacity and throughput because there are more disk spindles in the RAID disk set. The minimum RAID 0 set contains 2 drives. Some external home use drives use RAID 0 to increase capacity in which I DO NOT recommend unless you have another set of backups somewhere else and capacity is paramount. In my book, capacity never trumps reliability when it comes to data storage.

RAID 1
This RAID level known as disk mirroring (without parity). Simply put, RAID 1 is a duplicate image of the main disk on another disk. This is also called duplexing. This RAID level is usually done using hardware controllers, but can also be done using some operating systems that support disk mirroring or third part software. There are definitely advantages for using this RAID level, but is the most costly because essentially you will be buying double the usable disk capacity.

RAID 5
This RAID level is disk striping with distributed parity. What this means is that the data is distributed along with the parity data across all drives in the RAID set. This RAID level can tolerate a single drive failure which will need to be replaced. Upon replacement, the new drive must be rebuilt from the surviving drives. During this time, the failure of a second drive will result in data loss.

RAID 6
This RAID level is disk striping with double distributed parity. What this means is that this RAID set can tolerate 2 failed drives and still be operational. Failure of a third drive will result in data loss.

HOT SPARE
A hot spare is usually a disk that is powered on and spinning that sits in an array that does nothing but wait for a drive failure. The hot spare is then automatically rebuilt from the surviving disks in the RAID set . This will allow for minimized windows of exposure to data loss.

I will be expanding on RAID technologies in my future posts and hope this was helpful in understanding this complex but compelling technology.

It is amazing how our lives revolve around our computer(s). I actually felt like I was helpless when I left my laptop at home for a vacation. Of course, there was my blackberry that I was still able to get my fix of information. The days when all my worldly information could fit on a single 5.25” floppy disk are long gone. Like many people, you probably make backups of your data to an external hard drive, but is that enough to adequately protect your valuable data? How about all those tunes and movies that you spent a fortune on and all those years of digital pictures? What about your iPad or tablet? Are those protected from data loss? The other question is, what if I lost my portable device or portable hard drive?

I have so many friends that come to me with their portable hard drives that stopped working because they either dropped it or it just croaked. The fatal mistake most people make is when they start MOVING data to a portable hard drive instead of a BACKUP. That is when only 1 copy exists. I try to help my friends with some recovery techniques, but sometimes the portable drives are beyond my tools and abilities. Usually, at that point, some serious $$$ are needed to have the data recovered. How can you protect your valuable data from different levels of threats?

1. Backup, Backup, Backup!
Backup to an external drive. Then backup to another external drive. That way, you have 2 copies in an external source. Make sure if you move your data from your computer, keep another copy on another drive. If you have super sensitive information, it is a good idea to encrypt your external drives. That way, if you loose the drive, your data is unreadable. The other consideration is using a RAID technology in an external drive. These are more costly, but by RAIDing your drives, it adds protection from drive failures at the drive level.

2. Cloud Backup
Backing up the the “Cloud” is a great way to add extra protection to your data. Cloud backup is using your Internet connection to upload your data to some off-site location. Not all cloud backup services are the same. Make sure you do your research carefully to see if the service provides you with the security, capacity and cost you are comfortable with. Sometimes recovering to a USB drive or DVD is costly, so check before you commit. I like BackBlaze and Mozy for they are easy to use and reliable.

3. Use a MobileMe
I am not one to promote MobileMe because I personally think this is an expensive service and the cloud backup space is minimal. The one thing I really like about MobileMe is that you can locate your iPad or iPhone remotely as well as issue a remote “wipe” of your data if you ever lose your device. As for Droid devices, I am sure there either are or soon to come a similar service.

There are certainly additional ways you can backup your data, but I believe if you do follow these 3 steps, you will be protected pretty well.

Don’t wait till it happens, do it today!

I remembered when I got my first computer. It was a Timex Sinclair.

Yes, the watch company and they made a computer. It had about 2k of RAM with a flimsy cellular buttoned keyboard. It had no hard drive or floppy disks, but it had a cassette tape player as the storage device. That’s right, the good old cassette tape. I bought a game on cassette tape with the computer back in the 80s and it took about 15 minutes to load the game. It was painful, but worth it.

From there, I got a Commodore Vic20 with a 5.25” floppy drive. Wow, this was fast! From cassette to floppy disk seems like years ahead. I even used a hole-puncher to notch the side of the disk so I can use the flip side of the floppy. We didn’t have that many choices back then. Most games and applications loaded and ran right off of removable media.

I remember when hard drives first came out. It was very expensive and out of reach of the masses. A few years later, I saved up enough money to buy a 200MB hard drive by Western Digital. I paid $400 for that hard drive. It was worth every penny.

Since then, I was always impressed by how manufacturers were able to get more density on each disk every 6 months. Now with 3TB drives coming out soon, it never ceases to amaze me.

It is interesting how storage is going in 2 directions, Solid State Disk (SSD) and Magnetic Hard Disk (HD). If SSD is the next generation, then what does that mean for the HD? Will the HD disappear? Not for a while… This is because there are 2 factors: Cost and Application.

At the present time, a 120GB SSD is about the same cost as a 1TB drive. So it makes sense to go with a 1TB drive. Although this may be true from a capacity standpoint, the Application ultimately dictates what drive(s) are used to store the data. For example, although a 1TB drive may be able to store video for a video editor, the drive may be too slow for data to be processed. Same goes with databases where response time and latency is critical to the application.
Capacity simply will not cut it. This is when using multiple disks in a RAID (Redundant Array of Independent Disks) set will provide performance as well as capacity. SSDs can also be used because the access and response time of these drives are inherently faster.

As long as there is a price to capacity gap, the HD will be around during that time. So architecting storage today is a complex art for this dynamic world of applications.

I hope to use this blog as a forum to shed light on storage and where the industry is going.