Exploring swatch

First install swatch. It  is distributed through sourceforge as a source code release in a tar.gz file. You may find it in rpm form for your distribution. For Fedora, just try:

yum install swatch

Per the readme, "For a simple demonstration type 'perl swatch --examine=FILENAME' with 'FILENAME' being the file that you would like to see the contents of." I suggest /etc/resolv.conf because it is short. Take a look at it:

cat /etc/resolv.conf

then run swatch on it. The swatch perl script should be in /usr/bin. If it has executable permissions you can run it directly, instead of running perl and telling perl to do so. Give it executable permissions, if it does not have them, with:

chmod +x /usr/bin/swatch

Now run as either:

swatch  --examine=/etc/resolv.conf

or equivalently:

swatch  -f  /etc/resolv.conf

Because there is no configuration file in place yet, swatch complains. It's looking for the default, which is ~/.swatchrc in your home directory. Create that file, giving it these contents:

watchfor  =  /.*/    
        echo

This is a typical pattern-action configuration entry. Such entries tell swatch that, as it traverses down through the target file, it should look for certain lines and whenever it finds one should take a certain action. The regular expression following the watchfor directive tells what lines to look for. Here, that expression is  .*  (dot star). It signifies any line (dot means one character of any value, star means zero or more of the preceding character; dot star means zero or more characters of any value). The echo keyword signifies what to do when one of the specified lines is found. echo means, print the line on the screen. So the entry as a whole says, "print all lines on screen." After swatch's own messaging appears, you should see the same recapitulation of /etc/resolv.conf on screen as you did with cat.

Now let's create a configuration file to do something a little more realistic. Instead of /etc/resolv.conf, it's common to target /var/log/messages. This is the central log message repository used by many applications. While applications may maintain their own separate log files, many adopt /var/log/messages. Swatch, which looks at whatever file you tell it, can be trained upon any of these.

When he uses swatch, the administrator knows in advance what kinds of messages might appear in the log file. What he wants swatch to do is notify him or take other action when that happens. So he gives swatch enough information about the interesting messages for swatch to identify them, and tells swatch what to do when they appear.

For example, we know that when you try to log in but fail, a corresponding message is written into /var/log/secure, similar to:

Oct 6 12:20:15 EMACH1 login[2149]: FAILED LOGIN 1 FROM (null) FOR tom, Authentication failure

Make sure such a line is present in your "secure" log by performing a failed login. You might select an unused virtual terminal (e.g., ctrl-alt-F2) to see a login prompt at which you'll be able to fail. Give the name of a nonexistent account or use an incorrect password. Then return to the virtual terminal from which you came. We'll make a configuration file telling swatch to pick out lines like the above. "FAILED LOGIN" is a good enough as an embedded signature we can match. So, use an editor to create a file in your home directory named swatchconfig.one containing:

watchfor = /FAILED LOGIN/
        echo

Now run swatch, calling its attention to this as its config file:

swatch  -c  ~/swatchconfig.one  -f  /var/log/secure

Suppose you only want to see the login failures of users other than root (or some other user whom you know to have committed login failures). Elaborate the config file to read:

ignore = /FAILED LOGIN.*root/
watchfor = /FAILED LOGIN/
echo

There's the dot-star again. Any line containing FAILED LOGIN but with "root" showing up anywhere farther down the line, matches the regular expression in the ignore directive. That causes swatch to ignore the line (do nothing in response to its presence), even if this line would match subsequent directives that do call for a reaction. For this line, swatch never gets that far. It stops processing this line and moves on to the next. The first directive the line matches determines the reaction, trumping any matching directives that follow. Running swatch now produces the same output as before, but with the conspicuous absence of the lines from root's login failures.

It's great we apply this tool to a file like a stethoscope, follow its pulse, and react if we hear certain symptoms. But for this to work we have to know 1) what to look for, and 2) what to do. As a doctor needs to know both symptoms and treatments. In advance. Swatch will only react to what we've told it to, via the configuration file. So in order to write that file meaningfully we must know what's coming. From experience or experiment, we need to know the wording of the log messages for events we want to detect. Of course in the first place, there must be such messages and some software agent other than swatch injecting them into the log file. For example if you want swatch to page you whenever somebody does a port scan on your machine you have to run some software designed to watch for port scans. And that software has to talk to the log file as part of its reaction when a port scan happens. Swatch doesn't watch for port scans, it watches for log messages. But if something that does watch for port scans reliably issues corresponding messages, swatch is positioned to watch for scans in effect, indirectly. Works! Swatch can be the lookout for anything that gets logged-- port scans, bad logins, whatever. It just has to get logged.

Once you have swatch set up to react to some kind of event, it will react to its every occurrence. That's often what you want, but not if the event is happening too fast. Then you might want just one notification expressing the thousands of packets in a denial of service attack for example, not thousands. Swatch has a "throttle" notification-limit feature for that.

Let's create an artificial example. We want to create a storm of network packets, then set swatch up to react not to each one of them but to the storm as a whole. We need a tool to create the storm. We need that tool or some other to inject messages about it into the log file. And we need to tune swatch to those messages. The first tool will be the ping utility, and the second will be iptables, which is linux's firewall utility. ping itself doesn't routinely log anything. iptables on the other hand can do that job. Its primary job is network packet observation. You can tell it to look for certain types of packets. When it sees them, you can tell it what to do (often, discard them). But another thing you can tell it to do with interesting packets is log them.

Here's what iptables wrote in my log file /var/log/messages to represent 2 different packets that came along:

Oct 12 13:39:50 EMACH1 kernel: IN=eth0 OUT= MAC=ff:ff:ff:ff:ff:ff:00:40:05:a3:42:26:08:00 SRC=192.168.3.28 DST=255.255.255.255 LEN=340 TOS=0x00 PREC=0x00 TTL=128 ID=33015 PROTO=UDP SPT=68 DPT=67 LEN=320

With this knowledge of what there is to look at, we can define what we want to look for. You usually look for a uniquely distinguishing embedded string. For this artificial example, let's enlist iptables' help to deliberately embed something unique into all the messages it logs for us. It has a command option to add an arbitrary prefix to the messages it writes. We're going to tell iptables to log network packets bearing our unique prefix, but the network might be busy and we don't want to choke up our log file. So since we're planning to have ping generate the test packets, let's tell iptables to only log those ping packets. Please issue the following command:

iptables -A INPUT -p ICMP -j LOG --log-prefix 'TELLTALE ACTIVITY! '

A sample message thus logged appears below. Note the embedded "TELLTALE ACTIVITY!"

Oct 12 17:47:39 EMACH1 kernel: TELLTALE ACTIVITY! IN=eth0 OUT= MAC=00:c0:4f:5e:35:69:00:a0:c9:1a:a8:96:08:00 SRC=192.168.3.2 DST=192.168.3.12 LEN=84 TOS=0x00 PREC=0x00 TTL=255 ID=37514 PROTO=ICMP TYPE=0 CODE=0 ID=58397 SEQ=

Be aware that iptables' LOG target utilizes a program called rsyslogd, which is the actual writer of the /var/log/messages file. When iptables wants a message logged into /var/log/messages, it passes the desired message to rsyslogd and the latter writes it into the file. Make sure rsyslogd is running by executing:

service rsyslog status

(If not try to start it with service rsyslog start.) Another dependency is a possible pre-existing firewall. Such a firewall could effectively prevent the above iptables command from working. Your lab computer should be firewall-free. Check by executing

iptables -n -L INPUT

(If you see evidence of firewall rules you can flush them with iptables -F INPUT. If you do that, then afterward repeat the above "TELLTALE..." iptables command to reinstate the rule it provides for this exercise.) Now let's run our artificial example. Change to your home directory. Operating in the GUI, please open 2 terminal windows. In window A please execute:

tail  -f  /var/log/messages

In window B ping to another machine on your network (this example uses 192.168.3.2):

ping -c 2 192.168.3.2

Repeat the above ping a couple times. Each time, ping reports how many packets (ping replies) were received. Meanwhile in window A the "tail -f" command gives you a dynamic "live feed" of whatever is being added to the log file. Observe both windows as you issue the above ping. Notice that messages logged in response to our ping contain our signature phrase "TELLTALE ACTIVITY!"  Terminate the "tail" command with a ctrl-C keystroke.

Now set up swatch to do much as tail just did, and take over the vigilance job overseeing /var/log/messages. Use an editor to create file swatchconfig.two with these contents:

watchfor /TELLTALE ACTIVITY!/
   echo

Then, in the window where "tail -f /var/log/messages" previously ran, launch swatch:

swatch -c ~/swatchconfig.two  -t /var/log/messages

Notice we are using the -t ("tail") option this time instead of -f, to get ongoing dynamic oversight instead of a one-time pass through the file. Again:

ping -c 2 192.168.3.2

observing both windows. Note that tail is limited-purpose but swatch is general-purpose. Tail just regurgitates on screen whatever log messages show up in the file. Here, swatch is doing the same thing but only as a special case, since we're using swatch's "echo" keyword. If you want to see a different response, you could replace "echo" with "exec echo BINGO!!" in swatchconfig.two and "BINGO!!" would appear on the screen to represent each ping log message, instead of the message itself. More practically, the "mail" keyword often used to send the log message to an administrator via email. And, with the exec keyword any arbitrary executable can become the response.

Now let's get serious. Run a hyper-ping in window B:

ping -w 1 -f 192.168.3.2

-f means a flood, or rapid fire, ping. -w 1 restrains it by holding the duration down to one second. In one second my machine can fire off some hundreds of pings. Each provokes swatch's response, showing up in window A some hundreds of times. If swatch provided for emailing the administrator, he'd get hundreds of emails. A denial-of-service attack with millions of packets could seriously clog his mailbox. He would probably be content to get just one email about it, or at least a mere fraction of the number of triggering packets. Modify swatchconfig.two:

watchfor /TELLTALE ACTIVITY!/
   echo
   threshold track_by=message,type=limit,count=2,seconds=3

then restart swatch. Now repeat the flood ping from the other window. How many swatch response messages show up in window A? Now run ping a controlled number of times over a controlled period, and note the number of  response messages. You can tell ping how many requests to issue with the -c (count) option. And you can specify at what interval with -i. Run 75 pings spread over 7.5 seconds:

ping   -i 0.1  -c 75   192.168.3.2

How many swatch response messages appeared? Why that number?

Finally for fun, get swatch to send you some email just like in the real world! Create swatchconfig.three:

watchfor /TELLTALE ACTIVITY!/
    exec /bin/echo "There has been a ping!" | /bin/mail -s "got pinged" <your account>@<your mail provider>

Make sure the mail agent is running

service sendmail status

And you know how to start it if not. So now run swatch:

swatch -c ~/swatchconfig.three  -t /var/log/messages

then send yourself some mail:

ping -c 2 192.168.3.2

Terminate swatch. Clean up:

iptables -D INPUT -p ICMP -j LOG --log-prefix 'TELLTALE ACTIVITY! '

or more brutally but simply

iptables -F INPUT

rm ~/swatchconfig.*