syslog exercise

Exercising syslog locally

While the syslog daemon can be launched directly (as /sbin/syslogd) there is a startup script for it. Use the script to check that the daemon is up:

service  syslog  status

(In case it isn't, rerun the script with "start" instead of "status.") The daemon accepts log messages and dispatches them to their destinations. It accepts them from sending applications; applications take the initiative. Once it has them, it dispatches them to destinations determined in its configuration file, /etc/syslog.conf. The default configuration file writes messages to various files, depending on the message. Examine one of the critical lines determining messages dispatch:

grep  "/messages"  /etc/syslog.conf

The right-hand field defines the file in which to write the messages of the type described by the left-hand field. That file is /var/log/messages, which is the most central and heavily used log file on most systems. You control messages' destinations by manipulating the right-hand field. Examine a sampling of logged messages:

tail  /var/log/messages

If you want to test syslogd, you've already got a message receptacle so now you just need to generate a message. For that you need to write or run a program that explicitly engages in sending messages to syslogd. Many do. If you were writing your own, you would incorporate library calls provided for the purpose. That's how it's done. On the other hand if you just want to observe the mechanism, simply run some existing program that throws messages. In the GUI, open two terminal windows, one to run commands and the other to watch the log file. In one window:

tail  -f   /var/log/messages

This is the log-watcher window. Leave this command running. It will dynamically update the screen with logged messages as they arrive. Then in the other window:

service syslog restart

Note that when syslogd stops it "says so" in the log file. Similarly when it starts up again. Another command that provokes log messages is su, used to switch user. Try it. You can see the message action in the watcher-window. There is a command, logger, for the specific purpose of giving users (and scripts) a command line tool for shipping a message to syslogd:

logger How now brown cow

Now that you've observed how syslog works, return to the first window and terminate the "tail -f" with a control-C keystroke.

Sending log messages to another machine (loghost) for disposition

Syslog gives machines a way to self-track their activities. Sometimes it's desirable for a machine to receive and track the activities of other machines. A machine that does that is called a loghost. Syslog can be configured to be a loghost (receiver), or to send to a loghost. Remote logging utilizes a network. A log host "listens on port 514." That's something that syslogd doesn't normally do. So turning a machine into a loghost is a matter of telling syslogd to pay attention to the network and listen for any messages that might appear from other machines.

Work with a partner. One of you will become a loghost, the other will send its messages to the loghost by designating the loghost by IP address as the message recipient. If you ran syslogd directly by running the binary, its -r option is the switch that turns on loghosting. "This option will enable the facility to receive message from the network using an internet domain socket.... The default is to not receive any messages from the network," according to the syslogd man page. However if you start syslogd from the startup script ("/etc/rc.d/init.d/syslog" or equivalently "service syslog start"), you tell the script to use -r via its sysconfig file /etc/sysconfig/syslog. Edit that file. Find the line that specifies the options to be used and change it by adding -r so that it reads:

SYSLOGD_OPTIONS="-m 0 -r"

Then restart syslogd:

service  syslog  restart

to make it adopt the new setting. It's now a loghost. So should some other host send a message its way, it'll accept and process that message. Sending something is the other student/partner's role.

Sending messages to a loghost is a matter of doing so instead of writing it to the usual local file. So the change (on the other machine!) needs to be made in /etc/syslog.conf. There, find the line that has "/var/log/messages" as its destination. Change that to:

@<remoteIP>

referring to the IP address of the partner who has just become a loghost. Then restart syslogd:

service  syslog  restart

Now, when this host sends a message, it will be to the loghost. What will the loghost do with it? Whatever its syslog.conf say. By default, it should put it into its own /var/log/messages. That file will become mixed now, some of its messages originating locally but others from a different machine. Set the loghost, with tail -f, to dynamically display activity in its main log file. Then have the other machine issue a message, perhaps using logger. Sites sometimes set a particular machine to server as loghost for a whole population of other machines. This good for security since an intruder who gains control of a machine does not gain control of its logged "history." Intruders sometimes falsify the log file entries that betray their entry, but now that information is safely out of the bad guy's reach.

After you have performed remote logging, reverse roles. Undo the changes you made on your box, and apply those your partner applied to his. Remember the operative files are /etc/syslog.conf and /etc/sysconfig/syslog.

Making remote logging secure

A possible concern with syslog's native remote logging is that it sends inter-machine message traffic in the clear, legible if intercepted. Moreover, its protocol for doing so is the simpler UDP not TCP. UDP is less amenable to being tunneled via other, encrypting tools. One solution is a project called syslog-ng that replaces syslog and adds features. Short of that, here is a homemade method that is crude and bears some refinement, but gets the job done while showcasing some other syslog and linux features.

Here's the strategy. We will modify the sender so it puts messages into a container, then write a custom program to take them out and pass them to ssh. All this on the sender machine. ssh in turn will talk to the loghost (more precisely, to sshd on the loghost) and tell it to execute a command that writes the message into loghost's log file. Note that the instance of syslogd on the loghost is not involved. The message "goes around" loghost's syslogd and sneaks into loghost's log file directly. The only service we need running on the loghost is sshd. The "container" on the sender will be a named pipe, or fifo. Se we need to learn about those. And the command that sshd can use to write messages into loghost's host file is a redirected cat (cat foo >> file).

Ingredient: named pipes

A named pipe is a special kind of filesystem element. You can see it in the filesystem using the ls command; it looks like a file. It serves as a kind of asynchronous pipe, for use by unrelated processes. A process writes something into it, another takes it out. The "deposit" and "withdrawal" can be at different times and the processes can come from different shells. This is unlike a regular pipe, whose "in" and "out" are expressed inseparably in a single syntax construct ( "process1 | process2" ) and whose behavior is immediate. When data is withdrawn from a named pipe, it emerges in the same order in which it went in. That is, First In First Out, so named pipes are also called fifo's.

For its part, syslogd knows how to send its received messages to a named pipe, as opposed to a file or network loghost. From there the messages are in reach of any arbitrary program that wants to come along and pick them up. Then that program can further process them as desired. (In this case, "desired" would mean having ssh shoot them over to the loghost and inject them into the log file there.)

To use a named pipe you must first create it. Then you put something in. Then take it out:

mkfifo /tmp/testfifo
echo love is blue >> /tmp/testfifo &

You'll get some information on the screen then the prompt. At that point:

cat /tmp/testfifo

You wrote the pipe with "echo >>" and read it with "cat."

Write two scripts, names "fifoin" and "fifoout" in the current directory. fifoin will put stuff into your named pipe every second:

#!/bin/bash
while true
	do
	echo -n "hello " >> /tmp/testfifo &
	sleep 1
done

While every 5 seconds fifoout takes out the accumulated stuff"

#!/bin/bash
while true
do
	cat /tmp/testfifo
	echo
	sleep 5
done

Make these 2 files executable (chmod +x), launch fifoin in the background ( & ), then run fifoout:

chmod  +x  fifoin  fifoout
./fifoin  &
./fifoout

Once you've seen your fill, terminate fifoout with control-C. Don't forget, fifoin is still there in the background jamming stuff into the fifo (see it with "jobs"). So bring it back to foreground with the fg command then terminate it too with control-C.

Writing the fifo with syslogd

We can continue using the same reader (fifoout) but we're going to switch writers now. Instead of fifoin, which is artificial, we'll make syslogd our pipe-writer. For syslogd, sending messages to a fifo is a matter of doing so instead of writing it to a local file or network loghost. The change needs to be made in /etc/syslog.conf. There, find the line that has/had "/var/log/messages" as its destination-- the one you changed earlier to your partner's loghost, the first non-comment. Change the destination field (right side) to:

|/tmp/testfifo

Then restart syslogd:

service syslog restart

Arrange 2 open terminal windows open in your GUI. In the first, set a guard on the named pipe:

./fifoout

Then in the second window stir up some syslogd activity:

logger Where seldom is heard a discouraging word
su root -c "service syslog restart"

Watch the show in the first window. You're seeing whatever people are putting into the fifo. Terminate fifoout in the first window.

Making the fifo-reader relay log messages through ssh

We've achieved the goal of getting syslogd's messages into the hand's of a general-purpose script, where we can program the messages' fate as we like. So let's change fifoout so that instead of cat-ing the messages to our screen, it sends them to ssh, telling ssh to ship them to the loghost and trigger the command over there to cat them into the remote logfile. We already did remote logging using syslog's native facility. This new way will be not only remote, but secure too. Edit fifoout to read as follows:

#!/bin/bash
while true
do
	cat /tmp/testfifo | ssh root@<remoteIP> "cat >> /var/log/messages"
	echo
	sleep 5
done

If you have made key pre-arrangements between the local user account you're using and the root account on the other machine (your partner's), this will go through without intervention, otherwise will probably ask for the remote root's password. Have your partner put up a dynamic watch on his messages file (tail -f). Then in one of your windows run fifoout and in the other stir up some trouble, just as you did above. Your partner's window should display the addition of your messages to his messages file.

I should add that there are better languages than shellscript for manipulating files, especially fifos. Shellscript lacks the precise character-level granularity of most languages' separate calls to open, move pointers, read, write, and close files. Nevertheless this exercise demonstrates an architecture and approach that gets the job done. Another alternative would be replacing syslog with syslog-ng, which addresses a number of syslog design design and feature limitations. Among them is secure remote logging.