Monday, November 28, 2005

Back it up...

Having backups is something I am totally into.  Remember though, it is the ability to recover that counts – having backups is not enough.  I’m not just talking about database backups, but rather file system backups of your computer.  How many of you backup your desktop/laptop on a regular basis?

The reason I’m bringing this up is because last weekend my son’s hard drive started failing.  He let the chkdsk run over night only to find out in the morning it had sort of “erased or otherwise made inaccessible hal.dll”.  That is a rather important file for getting windows to start.  He was a little bummed as his schoolwork was on there (yes, he backed it up – onto a pendrive, which he could not find – back to that “it is the ability to recover” comment.  Further, the backup was out of date).

So, I took the disk into work and borrowed an external USB enclosure for the little IDE drive.  Fortunately we were able to get the homework and some other data off of there.  Now he too has a 300GB drive as do I (well, I have more than one) and my wife.  He’ll be backing up more often to a drive that is just too large to “lose”.

Anyway, I’m always looking at how I backup.  I’ve written before about Unison, a freeware tool I’ve been using for a long time.  With that, I backup my really important stuff to three different machines – two at home and one at work.  In a true disaster (I’m on the road and lose my slides) I can always pull from one of those sources assuming I can get on the network.

With the addition of the 300 GB hard disk though, I’ve added another layer.  The disk I bought came with “bounce back express” which was adequate (I upgraded to “professional” to get some more options).  That does a full disk scheduled backup and can do disk synchronization in a manner similar to Unison (but nothing has done the synchronization as well as Unison so far…)

Over the last couple of days – I started playing with something else all together though.  A more real time sort of solution from NTI that permits versioning as well (you have to like that price until the end of December).  As I open and close files – this runs in the background and makes a copy to the external disk.  As someone who writes a lot – I think I’m going to like this (especially the versioning ability).  How many times have I had Word refuse to open my file – having the last 10 saves of it would be a life saver at times.  And since my laptop only has an 80 GB hard drive – the 300 GB should be able to hold multiple versions of many files.

So, now the routine will look something like this… When I’m at home at my desk, I’ll virtually be mirroring my laptop in real time.  I’ll still use Unison to sync up between the three external machines from time to time (like right before I unplug from home to go on the road).  When I come back from being on the road – bounce back professional will sync the laptop drive with the external disk and the NTI software will keep me in sync from then on out.

I’m sure it will be a function of how you work as to whether some like the above could work for you.  For example, if I actually ran a database on my laptop – that would probably not work so well (the NTI software).  But when I’m at home, I run the database on one of my two linux servers – at work on some server there.  On the road, I keep my virtual machines on external USB drives (better performance having a drive for the OS and a drive for the database) and back them up manually (just copy the VM’s onto my laptop as a last ditch backup).  So, for me it looks like this will work.  

At the very least, the NTI software is a fairly neat idea – you can point it at only certain directories if you wanted, I’m doing the entire machine right now.


Anonymous Andrew Henry said....

I had a drive fail once and luckily I lost only 1-2 months worth of unimportant mail--had a bit of a heart failure when I thought I'd lost a few folders of photos, but managed to retrieve them from family members.

After that I seriously looked into backup/recovery on the desktop...all open source tools like 7-zip to TAR the files, command-line Gzip to *quickly* do a reasonable compress, then GnuPG to secure the files. Finally burnt the resulting GPG files to DVD with a copy of all the tools (signed with GnuPG) and a document explaining exactly how I did the backup and how to restore. One copy I kept at home, one in the bank safe (only do this once a year or so) and one in my parents safe--they live abroad. Total disaster recovery.

The problem is, to do this manually, and to ensure its secure/open source, takes *time*. There is no versioning and selecting new folders/files to backup is a pain as 7-zip relies on the files being specified in a text file (as would TAR for Win32 if you used that app. ).

I now use NTI Backup NOW which came with my laptop--it doesnt secure files and it's proprietary (will I be able to read those .NTI files in 50 years??), but it takes 5 mins to start a backup.

What i'm saying is that it's all a trade off between how much time and effort you want to put into it and how important the files are. I decided that doing all the open source-secure-readable in 50 years-thing was just too much effort.

Did the NTI software you use have encryption as well?


Mon Nov 28, 08:23:00 AM EST  

Blogger Thomas Kyte said....

the NTI software I'm using is a file mirroring - so I'm reliant on "OS security". It is just files on the mirrored disk.

No encryption, would have to have the OS encrypt the file system if I wanted that.

Mon Nov 28, 08:27:00 AM EST  

Anonymous Michael Olin said....

My current desktop machine has dual 400Gb drives that I have set up as mirrors from day one. I also use Norton Ghost to keep a pretty recent image of the small (30Gb) drive on my wife's laptop on my mirrored drives. She backs up her documents to a CD regularly but I, of course, have bind faith in the mirrored drives.

I know I should burn a copy of my documents, data, etc. to DVD and promise that I will this weekend (well, probably not, and when I lose something important, it will be entirely due to my own laziness, since I do know better).

I did pick up 4 large (1Gb) USB flash drives last Friday ($39/each on sale), so there is a relatively painless way to copy at least some of my files to offline storage. Of course, even the minimal effort required for that gets thrown into the queue behind a family celebration, one of my son's swim meets, and a day of abuse for myself and my other son as my cousin takes us to the Patriots-Jets game to see his Pats destroy my pitiful Gang Green (at least the tailgating before the game will be excellent!).

Shameless Plug unrelated to the topic, but 100% Tom:

Come spend the day with Tom at the NYOUG Winter Training Day on January 18th. See the NYOUG website for details

Mon Nov 28, 09:27:00 AM EST  

Blogger Connor McDonald said....

The main problem I have with home backup is that concept of complete recoverability. Sure I've got backups of files, and yes, I can explore them etc etc... but that's a far cry from actually restoring complete a full working windows system.

So all my plain files can come back, but it sure would be nice to (say) boot of a CD and restore the full OS partition from an external drive back to the hard drive.

If anyone's got any recommendations, please drop me a line.

Mon Nov 28, 09:44:00 AM EST  

Blogger Howard J. Rogers said....

Speaking as someone who has successfully managed to lose an entire forum due to having a backup whose precise location was something of a mystery at the time it was needed (and subsequently), I couldn't agree with your sentiments more.

I am (or was. I've stopped of late... but for how long?) forever installing different O/Ses on the desktop, and losing the little bits and pieces of vital stuff that Windows tucks away. Your mail settings, for example. The Outlook inbox. The Media Player/Foobar database. Phut! And then 'soddit!'

I also back up regularly onto DVD. Honestly, I do. And onto a network drive. And onto an external firewire drive. I've got more backups of the Dizwell Website than you can shake a stick at. Only one slight problem: there are so many of them, I'm not sure which is which, or where any of them are. I lost the bottom half of a Data Pump article once... do you think I could find the backup with the correct ending in place? It was in the pile of 50 DVDs somewhere.

I am better with other people's data, for sure. But mysteriously, the skills all desert me when evening falls and I travel homewards.

It's like accountants, I suppose: I never met an accountant who ever did a reconciliation of their own bank account!

Mon Nov 28, 09:58:00 AM EST  

Anonymous Anonymous said....

For backing up documents I've been, for the past four years, using a wonderful little utility called "Beyond Compare", by Scooter Software.

It allows me to synchronize a folder on my Windows XP machine with a copy on a 750 MB Iomega Zip disk, a 35 GB Iomega REV disk, or an 80 GB external hard drive. At work, I use network folders as well as a 1 GB USB flash drive.

There's no reason to think it wouldn't work with a CD or DVD as well, assuming you have Drive Letter Access software enabled (something like Roxio's DirectCD or Nero's InCD). I'm not in the habit of using these because of some very bad experiences in the past, though perhaps it's time to revisit them.

At any rate, you can have "Beyond Compare" perform size, size/CRC, binary or rules-based comparisons. You can selectively include/exclude files and folders. It's possible to write command-line scripts to automate the synchronization process, though I haven't tried doing this yet.

As for full system backups, at this point I'm running NTBackup and storing the images on the external hard drive and the REV drive. As I understand it, I would have to reinstall the operating system, then pray that I could run NTBackup to do a full restore. Given how I only have the one computer at home, I'm not too eager to try testing a restore. If it doesn't work, I guess I'm stuck rebuilding everything from scratch.

I've worked with Norton Ghost in the past, but was put off by its inability to exclude certain non-essential portions of my hard drive. At the time, I was using "Virtual CD" to copy various CD and DVD images to disk for faster access; I would have preferred to rebuild the images rather than try to back them up with Ghost. Also, I ran into some problems Ghosting to an external hard drive. Has Ghost improved?

Bob Shepard

Mon Nov 28, 12:24:00 PM EST  

Anonymous Anonymous said....

The idea of losing the data on my local hard disk drives gives me nightmares. There is too much data to be backed up on DVD disks. It will probably be more than 50 disks. The price of disk drives keep falling and the capacity keeps rising. However, there is no good way to back these drives up. I considered raid, but don't have enough physical space for the redundant disks. Wish there could be a better and inexpensive way to backup the gigantic ATA drives on home computers.

Mon Nov 28, 01:26:00 PM EST  

Blogger Joel Garry said....

I gave my old PC to my four-year-old. As I was cleaning it up, I removed quite a bit of stuff. At some point, it said no more programs use vmm32.vxd, do I want to remove it? Now windows won't start, need to either recreate the vxd file (now that I know what it is and found purported instructions... which seems to have some misinformation about the registry on first glance), get it off a tape backup (around 14 tapes per backup, and I don't really know if that would work after having modified the system), or reinstall Windows. siiiiigggghhhhhh.

A year or two ago I successfully tested restoring a 3 tier Oracle system via Ghost for the non-data parts, but never had a warm-n-fuzzy feeling about it.

Mon Nov 28, 02:05:00 PM EST  

Blogger Robert said....

I never persoanlly had disk failure.
Been taking regular backups until 5 years ago.
Since then I've been really slacking off and taking my chance.

Mon Nov 28, 03:15:00 PM EST  

Anonymous Anonymous said....

I use Cobian Backup ( ) and

Simple, automated and offsite.

Mon Nov 28, 03:45:00 PM EST  

Anonymous Anonymous said....

What works for me is:-

1. Get windows installed and all your programs and settings you like. Then create a ghost image of the whole thing - windows can be restored in about 10-15 minutes like this and be EXACTLY as you left it! Used it many times on many PC's!

2. Each day I have a windows batch job that reads a list of directories out of a text file and using command line winzip it archives them up, puts a date stamp in the filename and copies it off to a mapped network drive. I even used to this recoved files before so it works!

My batch job kind of slows the pc down a bit, but I just schedule it for 1.15pm when I should be on lunch anyway!

Mon Nov 28, 06:42:00 PM EST  

Blogger Thomas Kyte said....

what do you use to ghost? is that just the norton utility?

Mon Nov 28, 11:22:00 PM EST  

Blogger Howard J. Rogers said....

I realise the question wasn't directed at me, but I'd like to put a vote in for Acronis True Image. Does what Ghost does, only it does it whilst the PC is up and running and being used. Ghost used to require a reboot with a DOS floppy (interesting trick on a floppy-less laptop), though I seem to recall they bought out a company that had mastered the art of disk-imaging-whilst-in-use several versions before they did, and that the acquired technology now means even Ghost is usable whilst the PC is.

Tue Nov 29, 01:06:00 AM EST  

Anonymous Anonymous said....

Hello Tom

Totally apart from subject of recovery, in this specific case of "erased or otherwise made inaccessible hal.dll" I found another solution. Just go back on installation cd and find there the original file "hall.dl_" (after booting from cd in recovery mode) and just replace the bad one (using a command like "copy hal.dl_ "c:\windows\system32\hal.dll""! After this operation Windows will boot and will ask for all drivers to be installed. I assume that "hal.dll" is some file with a lot of drivers inside. It is a lot easier than reinstall Windows.


Tue Nov 29, 03:39:00 AM EST  

Anonymous Anonymous said....


Yea, I use Norton Ghost to image the drive. In the event of losing windows, you just use a special bootable floppy (the Norton utility allows you to create one of these - newer versions may have a bootable CD version, I am not sure).

One time, when I upgraded my hard drive, I was able to 'clone' my current drive onto a much bigger partition on the new drive so my C: drive worked just as before, only bigger - saved a ton of time.

I am not sure I fully trust the utilities that can clone you drive while you use the PC (backup files probably ok, but clone the whole partition?) - you are going to be backing up things that are changing etc so its a bit more risky - The last version of Norton Ghost I used (2003) allowed you to select what you wanted to do in windows, then you click a button and the PC restarts in a DOS like mode, does the work and then restarts again.

Tue Nov 29, 06:44:00 AM EST  

Anonymous Andy Todd said....

I backup using rsync to an external hard drive and a usb pen drive. I keep leaving the pen drive at work and for some reason some machines won't connect to my home NAS via samba. when my laptop died recently it took me forever to restore back into a usable state.

I then took the bit between my teeth and now have my entire home directory in an off site subversion repository.

Which is lucky because when I hosed my work laptop at the weekend all I had to do after reinstalling the operating system was a quick svn checkout and I was up and running again.

I'd like to take credit for this but I nicked the idea from Martin Fowler

Tue Nov 29, 07:10:00 AM EST  

Anonymous Anonymous said....

When I was in school, I could use the "dog ate my homework" line. Now it's the "dog ate my hard disk"

Tue Nov 29, 08:27:00 AM EST  

Anonymous Anonymous said....

Norton Ghost, at least what comes with Norton Systemworks Premier 2005, does not shutdown the computer for backup. For recovery, Norton supplies a separate Recovery CD. Boot from the Recovery CD and perform recovery. I have never tried recovery. The Norton Ghost will automatically split the backup into 4GB slices for copying onto a DVD disk if no size is specified. Another good thing is that once you buy any Norton product like anti-virus or Ghost or Internet security suite or Systemworks, you can buy subsequent yearly releases for free after mail-in rebates.

Now I am just waiting for Blu-ray DVD or HD DVD disk to backup my 200+ GB data.

Tue Nov 29, 08:32:00 AM EST  

Blogger Thomas Kyte said....

in this specific case of "erased or otherwise made inaccessible hal.dll"

Yeah, we tried that - problem was hal.dll was the first missing file, there were many many many more (disk was failing big time).

He was happy to have his homework back :)

Tue Nov 29, 09:30:00 AM EST  

Anonymous Colin said....

Anyone know of a good Open Source/Free Software solution for backup and recovery?

Tue Nov 29, 10:23:00 AM EST  

Anonymous Basil said....

I'd evaluate any product from NTI Software VERY carefully. I paid good money for Backup Now just a couple of years ago. It ended up being unable to restore ANYTHING it backed up to a series of CDs. All I got from NTI was a shrug.

Seriously, the product just didn't work. The money back guarantee they hyped only applied if you bought it directly from them. If you bought it at a local store, as I did, you couldn't return it if opened. And that all pales with a backup that's not a USABLE backup, which appears to be your point here anyway.

Tue Nov 29, 11:03:00 AM EST  

Anonymous Roy said....

Really interesting reading!

My hard disk (c:) failed a week ago (missing one windows file to start) but I fortunately had all doucments etc. on another physical disk d:. When I had reinstalled c: (yes, I am also missing a good way to back up the complete windows disk - I only had a backup from d: to c: on the same pc - who expects both disks to fail at the same time :-) I decided that I would make a dualboot to be able to make a "cold backup" (full file copy) of the c:. After installing the first windows and during installing the second windows installation the harddisk failed again and would not start the seccond installation. OK... I didn't mind that too much and continued installing some programs on c: then the power supply stopped! Now I am ordering new power supply, raid controller card and a couple of new disks... Will have hardware raid on c: and XP sofware raid on d:

My next pc will have raid on the motherboard I will take advantage of that!

Tom, can you say a few more words on what you ment by "On the road, I keep my virtual machines on external USB drives (better performance having a drive for the OS and a drive for the database) and back them up manually (just copy the VM’s onto my laptop as a last ditch backup)" ?

Tue Nov 29, 03:52:00 PM EST  

Anonymous Scott P. said....

I use an Open Source tool called Bacula to back up all my Linux and Windows machines including my wife's Law Office. It is a full enterprise class backup solution that can write to tape, autochanger, disk, dvd etc. The server will run on linux, solaris and a couple of other platforms but it has clients for Linux, Unix, Windows, BSD, Mac and others that support platform specific permissions and flags.

I have tested restoring and it works very well indeed as I have lost a couple of drives in the last while.

I currently only backup to disk storage but plan on implementing a once a week copy to several DVDs for truly off-site storage.

Tue Nov 29, 04:27:00 PM EST  

Anonymous Anonymous said....

Colin said....

Anyone know of a good Open Source/Free Software solution for backup and recovery?

Cobian Backup is free.

Tue Nov 29, 04:45:00 PM EST  

Blogger Thomas Kyte said....

Tom, can you say a few more words on what you ment by "On the road,

I have an external disk - really small footprint. On it are a bunch of virtual machines (VM's). They boot using vmware and have my databases - meaning, I don't have any database files (big files) on my laptop hard disk. So, I'm not using this to backup any database files (it wouldn't work very well, especially the NTI stuff!!). I do have a backup of my vm's on my laptop however that I refresh after big changes (like an install of a new database) on the external disk.

Wed Nov 30, 09:57:00 AM EST  

Anonymous Justin said....

Seen something like this years ago... our sysadmin told me he did nightly backups. I asked him if he'd tested recovery... he looked ashen for a moment - then argued with me about if that was necessary. Come to think about it, the firm went bust a couple of years ago...

Wed Nov 30, 10:10:00 AM EST  

Anonymous Anonymous said....

We've got heavy machines here (y'know, 120 cpu Sunfires with 12 power supplies, the city lights dim each time we do a power-cycle..) Anyway, backing up Terabytes of data to high-speed tapes via RMAN has always worked a dream.

But I've also experienced a few nasties whereby the restore/recovery is blocked because all the available tape channels are being written to (e.g. by MS Exchange)

So then there was a meeting with various parties attending. During the meeting the Tape Backup team announced that they had never really considered recovery in their plans.

The backup team leader has since moved sideways to a new position.

Thu Dec 08, 11:53:00 AM EST  


<< Home