Networks dominate today's computing landscape and commercial technical protection is lagging behind attack technology. As a result, protection program success depends more on prudent management decisions than on the selection of technical safeguards. Managing Network Security takes a management view of protection and seeks to reconcile the need for security with the limitations of technology.
This is, of course, the issue of this issue. The notion of novelty detection is not really new - it is rather like anomaly detection except of course that newness does not an anomaly make nor does an anomaly have to be particularly new. The idea of novelty detection is to find out new things that appear in your network environment. It stems from the notion that, in many cases, new things have a higher likelihood of being bad things than old things. This notion could be utterly false and likely is in many situations, but in those situations where newness translates into suspect, novelty detection can reign.
So let us suppose that you have a relatively stable network and you are looking for new things. What are you likely to find? Of course if there are activities like web browsing and emails underway you are likely to encounter every new association between a browser and server, pairs of emailers, and so forth. If you are in a small environment, this is not so bad. For example, I get a few hundred new addresses coming to my web site per day, and lots of return visitors. I get many thousands of new email address pairs per day, but they are almost all from spammers, and of course the vast majority of these are intercepted and discarded somewhere along the line.
The interesting things are novelties like sources of scans, sources of viruses, sources of strange and malformed packets, and so forth. The important things to know about these novel items is that once I have been told about them, I generally don't need to be reminded of them each time they show up. I would like statistics on them of course, how many over what period of time would be fine, and if they get to where they have made a substantial change in rates of appearance, that would also be good to know about, again a novelty.
It turns out there is a really good reason that novelty is a really good thing to have your computer systems seek out and report on. The really good reason is not that novel things tend to have a higher likelihood of being hostile, even if they might. It is also not that we don't need to investigate when we see things similar to those we have already seen, we do need to investigate them if we are to find out about them. The real reason to do it is that that's how our brains work.
Yes indeed. When we see the same thing again and again, we tend to start to learn to ignore it. There are many different terms for the different ways we ignore things, like aclimitation, adaptation, accommodation, ... seems like they mostly end in 'tion' and start with 'a'. Something about the action of not doing something. At any rate, our minds seek out novelty and place increased importance on novel things. This is also the basis for Shannon's information theory, which is the basis for efficient communications and Huffman codes and similar optimizations like space compression programs.
So our minds will seek novelty whether we want them to or not, and if we are to stay entertained with what we observe in our networks and continue hunting down attacks, it will be very useful in reducing boredom and increasing our efficiency if we can have computers help us find novelty - things with high information content.
One of the best ways to use novelty is in reducing spam. Unwanted email now apparently dominates Internet traffic. On my site, I periodically do authorized scans (don't do unauthorized ones please). These scans do things like test my change controls to make sure I haven't forgotten about undoing a temporary change, and they test my susceptibility to denial of services (I flood the wire with packets to see if my server will handle full bandwidth denial of service attempts). In my latest scans, we were unable to match the traffic volume of spam email with our intentional attempts to consume bandwidth. Yes, that's right. Spam email was so voluminous that it was a better test of our ability to handle denial of services than the standard denial of service tools in widespread use. My counters to spam will not be discussed further here, but suffice it to say that they are largely oriented toward reducing the bandwidth consumed, not just the bother of getting unwanted emails.
Novelty can be used to dramatically reduce spam. It turns out that a long time ago mailing list providers determined that in order to prevent massive email floods they had to confirm memberships. This is done automatically by sending out a confirmation email on attempts to join the list. You reply to the pseudo-randomly generated response email header and the list server then knows that you have a reachable email address and signs you up. The vast majority of spam has the characteristic of false email return addresses, and of course if the senders did answer replies, they would be flooded if all email reception systems sent autoreplies verifying their email addresses.
With regular email a very similar thing works. In this case, the first time you get email from a new correspondent, you send them back a confirmation message. It is nice and polite and explains that once they reply to this message they will eternally be able to communicate with you without added inconvenience. They hit the reply button once, hit send, and their mail gets through to you now and forever. On your side, if you get spam from an address, you can simply put the address on your evil spam list. This will cause responses asking for confirmation every time and never forward you any of the email, or if you want to be nice about it, it will simply accept and discard their emails or indicate that you no longer exist. The overhead is very small, the advantages are great, and it is based on novelty.
If it works for email, why not for everything else?~ The first time they show up at your web server, you give them your legal notice, and from then on, let them use the site unimpeded. This is often done with cookies or some such mechanisms, but it can be done with other tricks of the trade.
FTP service should be a good one. The first time they come into your anonymous FTP server with their email address as the password, you send them a confirming email that gives them a user ID and password to use from then on. It eliminates automated FTP access, will get rid of web-based FTP access systems, and still allow anyone who wants to to get an account (you can make it pend approval if desired).
The list is endless, and many of these things have been used before to great effect. But that's not all you can do with novelty - not nearly...
In most of my networks we assign IP addresses to hosts rather than use DHCP. This is because most of our hosts act as servers. As a result, when a new host shows up on the network, I want to know about it. But how can I tell? If the person placing it on the network is really cleaver then I will probably not know anything about it, because there will be no information generated from the system. But if they send even a single packet that is not perfectly spoofed, my mechanisms will identify the novelty - a new MAC address on the network, a new IP address, a new port for an existing address, a TCP sequence without a SYN packet, a web request without a GET, POST, HEAD, or other authorized protocol element, a large volume of GET requests from one place to another, you name it.
My novelty detector also distinguishes distant events from local ones. This is particularly important since events that happen outside of my physical space are treated as a lot less novel than those inside. After all, you can expect just about anything from the Internet, but from your own network, that's a different story.
And how do I tell what's from my network and not? I have a novel approach - I will leave it at that.
Well, I hope you haven't wasted another 15 minutes reading my novel (not) on novelty. My only conclusions here are that if you haven't tried novelty out, you should. If nothing else, it will be different from what you are doing now, and that should at least keep you entertained for a bit.
If you have tried novelty, you might try not using it for a while...
About The Author:
Fred Cohen is helping clients meet their information protection needs at Fred Cohen & Associates and Security Posture and doing research and education as a Research Professor in the University of New Haven's Forensic Sciences Program. He can be reached by sending email to fred at all.net or visiting http://all.net/