syslog exercise - featuring rsyslog

syslog is the longstanding unix daemon for centralized logging. In recent years some upward compatible but feature-extended replacement projects have appeared, most notably syslog-ng and rsyslog. For lack of acceptance as standards their adoption has been slow and syslog remains the most widespread logging daemon. However, recent versions of the Fedora distribution of linux have replaced syslog with rsyslog. rsyslog's features can be viewed as a superset of syslog's.

Exercising rsyslog locally

While the rsyslog daemon can be launched directly (as /sbin/rsyslogd) there is a startup script for it. Use the script to check that the daemon is up:

service  rsyslog  status

(In case it isn't, rerun the script with "start" instead of "status.") The daemon accepts log messages and dispatches them to their destinations. It accepts them from sending applications; applications take the initiative. Once it has them, it dispatches them to destinations determined in its configuration file, /etc/rsyslog.conf. The default configuration file writes messages to various files, depending on the message. Examine one of the critical lines determining messages dispatch:

grep  "/messages"  /etc/rsyslog.conf

The right-hand field defines the file in which to write the messages of the type described by the left-hand field. That file is /var/log/messages, which is the most central and heavily used log file on most systems. You control messages' destinations by manipulating the right-hand field. Examine a sampling of logged messages:

tail  /var/log/messages

If you want to test rsyslogd, you've already got a message receptacle so now you just need to generate a message. For that you need to write or run a program that explicitly engages in sending messages to rsyslogd. Many do. If you were writing your own, you would incorporate library calls provided for the purpose. That's how it's done. On the other hand if you just want to observe the mechanism, simply run some existing program that throws messages. In the GUI, open two terminal windows, one to run commands and the other to watch the log file. In one window:

tail  -f   /var/log/messages

This is the log-watcher window. Leave this command running. It will dynamically update the screen with logged messages as they arrive. Then in the other window:

service rsyslog restart

Note that when rsyslogd stops it "says so" in the log file. Similarly when it starts up again. Another command that provokes log messages is su, used to switch user. But it uses a different file to deposit its messages, /var/log/secure. Try it. After an "su" check for the resultant messages at the bottom of /var/log/secure. There is a command, logger, for the specific purpose of giving users (and scripts) a command line tool for shipping a message to rsyslogd:

logger How now brown cow

Now that you've observed how sryslog works, return to the first window and terminate the "tail -f" with a control-C keystroke.

Sending log messages to another machine (loghost) for disposition

rsyslog gives machines a way to self-track their activities. Sometimes it's desirable for a machine to receive and track the activities of other machines. A machine that does that is called a loghost. rsyslog can be configured to be a loghost (receiver), or to send to a loghost. Remote logging utilizes a network. A log host accepts log messages that show up on a designated port (traditionally it was UDP port 514). That's something that rsyslogd doesn't normally do. So turning a machine into a loghost is a matter of telling rsyslogd to pay attention to the network and assimilate any messages that might appear from other machines.

Work with a partner. One of you will become a loghost, the other will send its messages to the loghost by designating the loghost by IP address as the message recipient. (Well call the loghost the server, and the message sender the client.) If you run rsyslogd directly by running the binary, its -r and -t options are the switches that turn on loghosting using UDP and TCP respectively. On the server, turn off rsyslog:

service  rsyslog  stop

Now make the needed modifications in the /etc/rsyslog.conf configuration file. Note the following section near the top of the default file:

# Provides UDP syslog reception
#$ModLoad imudp.so
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp.so
#$InputTCPServerRun 514

The "im" in imudp.so and imtcp.so stands for "input module." If you load one of these, you are inserting in memory the code to make rsyslog attend to incoming UDP and TCP datagrams. Otherwise, it is dead to them. You can uncomment either the UDP directive pair, the TCP directive pair, or both. You can also control the port number here. Please uncomment the TCP directives and change the port number to 61514 so that the above snippet becomes:

# Provides UDP syslog reception
#$ModLoad imudp.so
#$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp.so
$InputTCPServerRun 61514

Restart rsyslogd:

service  rsyslog  start

syslogd is now a loghost, gathering up any log messages that arrive on TCP port 61514. Ask netstat to verify that:

netstat  -pant

A line showing "rsyslogd" and "61514" should appear in the output. So now, if another host sends a message its way, it'll accept and process that message. Sending something is the client student/partner's role.

Sending messages to a loghost is a matter independent of writing it to the usual local file. You can still write to the file, additional to sending to the loghost, or not. So the change needs to be made in the client's /etc/rsyslog.conf. There, find the line that has "/var/log/messages" as its destination. Change that to:

@@<remoteIP>:61514      (this, on the client)

referring to the IP address of the partner who has just become a loghost. @@ tells rsyslogd to have its stuff emitted in TCP datagrams, whereas a single @ would have it sent out in UDP datagrams. Then restart rsyslogd:

service  rsyslog  restart

Now, when this host sends a message, it will be to the loghost. What will the loghost do with it? Whatever its rsyslog.conf says. By default, it should put it into its own /var/log/messages. That file will become mixed now, some of its messages originating locally but others from a different machine. Set the loghost, with tail -f, to dynamically display activity in its main log file:

tail  -f  /var/log/messages

Then have the other machine issue a message, perhaps using logger:

logger  "This is a test"

Sites sometimes set a particular machine to serve as loghost for a whole population of other machines. This is good for security since an intruder who gains control of a machine does not gain control of its logged "history." Intruders sometimes falsify the log file entries that betray their entry, but now that information is safely out of the bad guy's reach.

After you have performed remote logging, reverse roles. Undo the changes you made on your box, and apply those your partner applied to his. You might try UDP, and some other port number, this second time around.