News and Views

Recently, nerds across the land have been all atwitter over what is fashionably referred to these days as an “epic fail.”  Google the subject if you want more details, but a relatively short form goes something like this:

The cell phone company T-Mobile put out a nice little smart phone called the Sidekick.  The Sidekick is possessed of no local non-volatile storage for productivity data.  (Translation:  if you turn it off your contacts, appointments, emails, etc… go away.)  Not a problem though; the phone connects to a bunch of servers by way of T-Mobile’s cellular data network, and these servers apply some think-balm to the Sidekick’s amnesia.

T-Mobile is not really in the business of running the kind of data center necessary to provide their smart phone customers with the “cloud” services required to make this all work.  They understandably handed off the responsibility of managing the services in question to a company called “Danger.”  (For those of you familiar with horror movies, this is the point at which the creepy music starts.)

Time passed and the Sidekick gained a loyal following of devoted customers.   Somewhere along the line Danger was eaten by Microsoft, the company’s new owner.  Then, just a little while ago as of the time of this writing, a decision was made to upgrade the network at Danger.

The upgrade, as you probably have guessed, is the epic fail to which I previously alluded.   Not only did the patient not survive the surgery, but the parties performing the upgrade were unable to restore the subscribers’ data for several days.    

Although Danger / Microsoft teams are “working around the clock” trying to recover the data somehow, T-Mobile has warned Sidekick users that some of them they may be out of luck.  It may prove to be too late for these unlucky folks, but there are lessons to be learned from mishaps such as this.

First of all, always have an easy way to keep an offsite copy of your data over which you have control.  This copying method is only of any value if you use employ it on a regular basis.  

In my case, I subscribe to a service called “Mobile Me”.  From my perspective, the relevance of Mobile Me is that it allows me to sync appointments and contacts from my IPod Touch with Outlook on my PC and ICal on my wife’s Mac.  It works wonderfully… most of the time.

Twice now we’ve had some issues where one or more of the devices in the mix would not sync anymore.   We called tech support both times and were able to speak to an actual human being fluent in the English language.   They were helpful and tried to debug the issue, but in both cases it required going to a backup.   

This might be the time to own up to a couple things.    First, I’m very apprehensive about upgrading my IPod.  Every single time I’ve done an upgrade on the device from a Windows machine it has required a full wipe & restore.  Not fun.  This is relevant given that the problems in question might have been avoided were I to have been running the most recent software available for the IPod.  The second thing bit of relevant trivia is that I make periodic backups of my own local data files; including those files used by Outlook.

The bottom line is that a simple restore of my Outlook data files fixed the whole problem, other than the fact that I lost a day’s worth of events.  I don’t know how well the backups at Mobile Me might have been in my case, because I didn’t need to use them.

Control over your backups is a big deal, and keeping a copy of your data out of sync with the daily grind of operations is about the most important kind of control that you can have.   The synchronization with Mobile Me is analogous to the kind of useful redundancy given on a network by the use of a storage technology called “RAID”.

“RAID” is an acronym that stands for “Redundant Array of Inexpensive Disks.”  The idea is that you setup two or more disk drives that function as if they were one drive.  If you are feeling really lucky and are the kind of person who would drive your new sports car in the snow at full speed, you can configure the drives to act as one big one.  This is more than a little bit risky since there is no fault tolerance whatsoever.

Most people choose to use RAID to increase their level of safety, rather than using it to merely increase the amount of storage that they have on a good day.  The single most common configuration is to setup a disk array (a group of disks) to do “RAID 1” storage.  A much more common name for “RAID 1” is “disk mirroring.”

When you configure an array of disks for mirroring, you pair the disks together.  The operating system on your computer then sees the pairs of disks as one functional unit.  Data is written to both disks at the same time.  This gives you a safety net should one of the drives fail.

A typical drive failure scenario where you are protected by “RAID 1” goes something like this:  most users on the network don’t care.  Everyone is puzzled by the fact their resident network geek looks a little stressed and is telling the boss that it is time to order a new drive for the server.    Understandably, this is far preferable to the alternative scenario in which the server reverts to a drooling dysfunctional heap of lost data.

Unfortunately, there are three inherent limitations to any “fault tolerance” technologies such as RAID.

1. Both drives in a pair could still be healthy, but useless if the server itself dies.
2. If the room or building in which the server sits is trashed, or becomes inaccessible, the server is frequently out of the picture as well regardless of whether it is still in working order.
3. Faithful and accurate copies of your data made in real time are only helpful if the current state of your data is actually something you want.  What happens if you just deleted a file?  It was just deleted on both of your mirrored disks.  Whoops.

The means by which these problems are addressed is conceptually simple, though sometimes annoyingly complex in practice:  you back your stuff up.

Backup copies are different from the copies created during the normal operation of a RAID array.   For these backup copies to be truly useful, a few simple rules must be followed:
 

1. The backup copy must be written onto media that is out of sync with the normal ebb and flow of file creation, deletion and modification on the network from which the original copy came.  If the backup can be easily modified by users on your network during the course of normal operations, it is not safe to consider it a trustworthy copy.
2. The media onto which the backups are written must be removable such that it can be taken offsite.
3. Multiple copies should be made.  A good rule of thumb is to have at least three copies- one offsite, one onsite and connected, and one onsite and disconnected, but available for quick connection should the need arise.
4. The backup process should be automated.
5. The media onto which the backup is written should be trivially accessible using commodity components that could purchased at just about any electronics store.
6. The format in which the backup data is stored should be easily decipherable.  Avoid proprietary archival storage schemes whenever possible.
7. The automated backup process should keep logs, and those logs should be checked periodically by an actual human being capable of reading and understanding them.
8. Test restores should periodically be performed, ensuring that the backup is actually a usable copy.
9. A full, trustworthy backup should be made of any critical systems prior to any major changes being made to the systems in question.  The backup media should then be disconnected from the system prior to the changes being made.
10. Copies should occasionally be pulled out of circulation for long term storage as a “moment in time.”
11. You should have a bare metal disaster recovery strategy in place.  Translated from Geek to English, this means that you have a plan for recovering from a total catastrophic failure in which you must start from scratch.  “Bare Metal” is a common nerd term referring to a new computer taken out of the box.

If your organization is one to which special data retention rules apply then your backup strategy must reflect this.  Do you need to make unalterable backups of your email every month in a format easily accessible to auditors?  Might you be served with a public request for information?  Make sure to know such things in advance of crafting your backup strategy.

Whatever you do, assume that you will eventually become a statistic.  A good rule of thumb to follow is that virtually 100% of all computer users will someday lose their primary copy of critical data due to hardware malfunction, operating system problems, network errors, or simple human mistakes.  If this rule applies to giants like T-Mobile and Microsoft, it definitely applies to you.  

In late 1998 I was at a nerd trade show that led to me getting the best value in technology that I have ever come by. It one was many shows I’d been to that year and not particularly remarkable in any way. I walked around and looked at new techno-baubles. I spoke to a bunch of really bored vendors with arthritic smiles and cheap suits. I also threw my business cards into the requisite contest fishbowls.

I get a massive amount of spam already, I like and generally trust my email filters and I filter fairly quickly by hand as well. As a result, I don’t think twice about tossing my card into a contest bowl at a conference. This is a bit of delusion on my part as I almost never win anything that costs more than about $30. The key word in that last sentence is almost.

Later that week, on the last day of the conference about half an hour before they closed, I got a call from a Digital Equipment Corporation representative. He was telling me that I’d won a computer. I thought it was a rubbish call at first, but decided to drive down to the Exposition Center that was, after all, only ten minutes away.

I had been instructed to go to the service entrance, give my name, the purpose of the visit and specify the person to whom I wished to speak. I had to do almost none of the above as I was greeted by the sales rep on the loading dock. He led me through a noisy maze of partially disassembled displays and sales people with loosened ties. I felt like the one salmon that had become confused and decided to swim back to the ocean.

After a few minutes of dodging burly guys carrying chunks of technology in large suitcases, we found ourselves at what was left of the Digital booth. “There ya go, Darrell.” The rep said to me as he gestured to a rather nice looking computer that was nearly their top of the line.

I was impressed. It was a very nice machine. I knew the specs and was shocked that they had actually given away something that good. “Do you need to see some ID?” I asked him.

He paused for a moment. “Yeah. Sure.” Before examining my offered driver’s license. “Looks good. Enjoy.” He then handed me the computer, shook my hand and waved goodbye.

I took the computer home to a wife who was flabbergasted when I told her what had happened. We setup the computer and used it for a few years; first as a primary system then as a secondary system. Finally, at an age of about five years of faithful service, we looked at the Digital and said “Wow. That thing is way too slow for what we do now.” It was time to hand it over to the house of the in-laws.

The computer then found its way into my wife’s homestead, into the room that once had been her bedroom and since has become a study. For the last five years it has been serving its new owners faithfully. In a couple weeks I will be replacing the old faithful Digital with another hand-me-down computer. This one will be a Pentium 4 HP that boots in about a fifth the time of the Digital and runs Windows XP. I think they will notice the difference, I can only hope that they will not notice a difference in reliability.

I find myself asking the question, what made this machine so good? What has enabled a computer to survive a decade of hard use?

For starters, it isn’t a cheap piece of plastic Chinese garbage where the only metal is the lead in the paint. Of equal importance though is the fact that it is very serviceable.

The Digital is on its third power supply. It is on its third hard disk as well as the result of one failure and one planned upgrade. Its memory has been expanded a couple of times in order to accommodate advances in operating system technologies and application software.

As I stare at the currently reliable and very useful, but largely unserviceable IPod Touch sitting next to my keyboard, I wonder what it will be doing in ten years? What is happening to quality in the technology industry these days?

The summer after my freshman year of college I took a full time job working as what could best be described as a techno-grunt. I did a tiny bit of programming, but mostly I carried heavy stuff from point A to point B, helped sling cable through ceilings, crawled through tunnels, repaired countless computers and terminals and burned myself with a soldering iron more times than I could count. I also helped with backups.

I and one of my friends with whom I worked at the time had the job of feeding the mainframe tapes as it went through its backup. The data on the staggeringly expensive cluster of hard disks (over 300MB of storage!) needed to be protected in case of some disaster.

At the time it took and average of fifteen 9 track tapes to backup the system. These were beasts that were about a foot in diameter and about half an inch thick. A full backup set easily weighed 40 to 50 pounds. Every week we would record a full backup and every month we would take the previous month’s tapes into a neighboring facility. We would travel on foot, with my friend and I sharing the weight of the month’s batch.

We were quite a sight walking between buildings with a half inch thick steel pole over our shoulders. On the pole would be about 60 of these tapes, dangling like some great plastic armadillo on a spit. It was fun in a way; mostly because it gave us a chance to see the light of day during working hours.

During my third year there, I was awarded a salaried position and made a manager. Around about the same time most of the backup went over to a new technology - DAT tapes. These little things could hold more data and moved like the wind even if they were really flimsy at first. The mainframe ate them for lunch; frequently getting them stuck badly enough to warrant a service call. It was still more efficient than the 9 track tapes, although I missed the excuse for fresh air.

It has been three years now since the last time I setup a server with a tape drive. These days, every time we put together a backup solution for our clients it involves the use of removable hard disks as the backup media. Every once in a while I will burn critical data sets to DVD ROM disks, just for that extra piece of mind. However I find myself perfectly comfortable using removable SATA disks for a massively fast backup solution.

As I watch SD memory cards and flash drive prices drop though, I wonder how long it will be until I hear a client say “Darrell, can we just back up our servers onto the flash drive that we bought at that office supply store down the street?”