Networks dominate today's computing landscape and commercial technical protection is lagging behind attack technology. As a result, protection program success depends more on prudent management decisions than on the selection of technical safeguards. Managing Network Security takes a management view of protection and seeks to reconcile the need for security with the limitations of technology.
I cannot express to you how painful it has become to do proper backups lately. Last month I bought 4 new 120 Gig hard disks. 2 of them were to backup the other 2. And I see no end in sight.
This month's article is about the increasingly decreasing options for doing backups and the implications of the lack of backup on the effectiveness of protection.
But before I go into technologies for backups, I think I should describe how I got into this mess.
Some years ago I decided to start burning CD-ROMs as a way to backup my systems. I had organized my systems so that the vast majority of user data resides in identifiable subdirectories, and this meant that I no longer had to back up the whole system in order to have a viable capability to recover from a serious problem. There would be a lot of configuration challenges once up, but other than some relatively minor inconvenience (e.g., temporarily not being able to print until I configured a printer), I would be back up and running in 4 hours or so. This was a tremendous advantage over the previous tape backup system which would take 16 hours just in tape restoration, followed by some complexity associated with having to take that backup versions of system files and meld them into the system that I had installed in order to do the restore.
The CD backups started me down the road of getting CD burners and CDs, which I now buy in volumes of several hundred at a pop (the CDs, not the burners which I only buy in volume 2-3 at a time). The problem with the CDs is their limited size. 640Meg is just not very much storage these days. It's fine for simple things, but it's not even big enough to hold that all.net web site, the vast majority of which I have personally written over the years (I give away a portion of it on my business cards, but the internal site is far far larger).
I am still looking at going to DVDs but the market is not quite right for it yet. It seems that every computer nowadays - or almost all of them - have CD readers, but not DVD readers. In order to do a restoration, I need to have a way to get the data in. Of course this problem led me to network-based restorations. This then means that I need to install an initial operating environment from a CD into the system to be restored then recover the backups from over the network. Naturally, this leads to the notion of doing the backup over the network as well.
So off I went to start network backups, and the first problem I encountered was those nasty firewalls that were designed to make me safe. The problem is that I either need a different backup system on every LAN I have or I need a way to move backups through firewalls. But of course the holes for backups are also possible holes for attacks. Yet the cost of an extra computer for every LAN for backups alone is a bit excessive in my view, especially since many of my LANs are temporary or relatively small. So I went to plan B.
Network backups are a huge advantage because all you need is one server with lots of disk space and it can be used to backup lots of content on lots of computers, restoring what is needed where it's needed when it's needed. I call it my 'spinning reserve', named after the motor generators that are always supposed to be spinning for extra power in the electrical power business. My spinning reserve consists of a set of fairly large (and ever-growing) disks containing current backups of all of my systems. Well, not quite really that, but close.
We still have that nasty restoration process, and did I mention spam? I'll get to that later - but back to the firewall. My firewall approach is simple. I make holes that allow me to do backups at specific times. For example, I have a monthly backup process with a checklist that I go through from start to finish. Somewhere in that checklist is the line that says something like 'enable backups from the server LAN' and it is followed by something like 'backup server 79' and that is followed by something like 'disable backups from the server LAN'. The enable and disable processes are essentially automated. I go to the firewall, enable the backup mechanism for the proper IP address pairs, do the backup, and when it is done, I disable that configuration line. This leaves a small vulnerability in that someone able to gain physical access to the server LAN might be able to interfere with the information exchange and insert or modify some content. Of course the content is moved using an encrypted authenticated TCP session via Secure Shell with server keys in addition to passwords required, so the risk is relatively low, but it exists nonetheless and I have decided to accept it until I have a better alternative.
I will discuss spam filtering in another article, but I wanted to mention that I backup all my email, and this of course means that the more spam I get the more storage of things I never really wanted in the first place I end up with. So I have taken an approach to spam that is a compromise. I use a spam filter to separate spam from non-spam on a statistical basis and, assuming my process works well enough, I end up with my directory full of spam at the end of every month. I back it up - yes - that's right - I back it up - but... I don't keep it forever.
In the case of spam and the backup copy of all incoming mail created before separation into span and not spam, I only save the last month's worth in spinning reserve. I know it is a lapse, but it accounts for something on the order of a DVD per month (4 gigabytes) and at this rate, it out paces the growth rate of hard disk drives. Rather than end up eternally increasing the number of disks in my spinning reserver, I have decided to get bigger disks at the rate of disk size increases and forgo saving all of the spam for posterity.
I have taken a prioritized approach to backup and restoration. This means that I have made explicit the priorities associated with backup and restoration of my information infrastructure. For example, if I have a major outage and have to restart everything from power down, I bring up the system I use most often (which also has the capacity to test most of the rest of the network), then the firewall (which is critical to communicating with everything else and finding out the situation), then systems ranging from servers in real-time applications to browser boxes used by visitors in roughly that order. Unless it is needed for some other part of the restoration process, the backup server comes up fairly late in the process. But if a recovery is needed, the spinning reserve takes top priority.
Like backup, restoration is done via a hole in the firewall, but unlike how I used to do restoration, I have decided that the bootstrapping process is simply too painful to be left to reinstallation processes. Over the last year or so, I have systematically moved toward a common operating environment (COE) for most of my systems.
In these COE systems, the recovery process is very rapid. It consists of generating a copy of a floppy, booting from a bootable CD-ROM, enabling the hole through the firewall, and doing a transfer of the relevant configuration and content information from the spinning reserve. In the case of systems like the firewall, a full restore takes under 3 minutes from power up to operation. For some of the larger servers, the server is up in under 3 minutes, but content returns over the period of an hour or more, starting with the most important content. In non-COE systems, restoration takes... actually, so far they have simply been put to rest and replaced with COE systems in every case when we have had a destructive crash.
Of course part of the reason I have all of this massive storage requirement for backup is that in order to get the COE environment working I have customized my bootable CD distribution. And if anything can eat disk space, is it having your own distribution of an operating system. It's not as bad as satellite imagery, but still it's pretty nasty.
To give you a sense of this, I currently have 7 iso images of CDs I am working on - that's about 5 gigabytes just for the images I am building and burning on a daily basis. The source tree for all of this includes 4 whole kernel sources and their interim compilation files and executable setups (several gigabytes), several partial pre-created sets of CD contents (several gigabytes), the source tree for all of these executables (another 10 gigabytes or so), and of course all of the special purpose software used to build and burn these various distributions. I save major versions, which means that I have 4 older incarnation copies of this whole setup and I am working on a next version over time. The net effect is that most of a 120Gig disk is consumed by this, with a second 120Gig disk on the same system as backup (it takes a while to save 120Gigabytes over the network and other users are impacted).
120 gigabytes is about 30 DVDs worth of storage, which means that even if I decided to burn DVDs as backups instead of CDs, it would still not be cost effective to move from spinning reserves to DVD-based backups. Except of course for the requirement that backups be separate and different - which is why I am moving toward DVD backups of the spinning reserves.
Backups have always had a fundamental problem in that they are never big or fast enough to make them efficient to use. As space requirements have grown, tapes have become increasingly ineffective, leaving only disk to disk backup over networks as an alternative. Backing up the backups presents its own problems and costs, and today we are pushing the edge of the capacity of even DVDs as backup media, and they have only become viable for wide-scale use in the last year or so.
This problem looks to get worse with time and for now requires management attention on the order of once per year at a minimum, just to create new backup strategies and acquire the necessary technologies to meet the ever changing situation. And yet doing away with backups is infeasible as it leads to loss of intellectual property which is the capital of the information age.
By the time you read this article, the details of my solutions will have changed yet again, but the problems remain the same. What, how much, what kind, how often, where, and by what process?
About The Author:
Fred Cohen is researching information protection as a Principal Member of Technical Staff at Sandia National Laboratories, helping clients meet their information protection needs as the Managing Director of Fred Cohen and Associates, and doing research and education as a Research Professor in the University of New Haven's Forensic Sciences Program. He can be reached by sending email to fred at all.net or visiting http://all.net/