|
|
8.7. LoggingIt's important to know what is going on on your system. Fedora provides a standardized, network-based logging system and tools to automatically monitor and trim logfiles. Understanding and using these tools effectively will allow you to keep your finger on the pulse of your system with minimal effort. 8.7.1. How Do I Do That?The syslog facility collects and routes messages in a Fedora system. The file /etc/syslog.conf configures the message routing; the default version of the file looks like this: # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log On the left side of each entry is a pattern that consists of selectors. Each selector contains one or more facilities (separated by commas), then a period, and then one or more levels (again, separated by commas). The facility indicates the origin of the log entry. Possible values are shown in Table 8-3. The level consists of a priority level and can be any of the values listed in Table 8-4, in increasing order of severity. Specifying a level means any message of that level or higher (more severe), so the selector kern.crit would match messages from the kernel with a priority of crit, alert, or emerg. To match only crit, an equal sign is added: kern.=crit. An exclamation mark negates a match: kern.!crit matches kernel messages with a priority below crit, while kern.!=crit matches all kernel messages except those with a priority of crit. An asterisk indicates that the facility or level should be ignored. Therefore, authpriv.* matches messages from the authpriv facility regardless of the priority, and *.info matches messages from any facility which are at the info level or higher. Multiple facilities or priorities can be matched using commas (indicating an OR operation), so mail,local3.* matches any message from the mail or local3 facilities. Multiple selectors may be included in one entry, separated by semicolons, which indicates an AND operation. The special priority none matches no messages from the specified facility. Therefore *.crit;kern.none matches all messages that are of crit priority or higher, unless they come from the kernel. On the right side of each entry in /etc/syslog.conf is a destination for the messages. The destination may be:
The order of the lines in the configuration file does not matter; every line is checked against each incoming message, so messages may be sent to multiple destinations. The default configuration file routes messages according to Table 8-5; as you can see, /var/log/messages is the prime source of information about the state of the system. 8.7.1.1. Interpreting /var/log/messagesThe /var/log/messages logfile contains entries similar to this: May 31 10:40:58 laptop3 dhclient: DHCPREQUEST on eth0 to 172.16.97.254 port 67
May 31 10:40:58 laptop3 dhclient: DHCPACK from 172.16.97.254
May 31 10:40:58 laptop3 dhclient: bound to 172.16.97.100 -- renewal in 34387
seconds.
May 31 20:14:05 laptop3 dhclient: DHCPREQUEST on eth0 to 172.16.97.254 port 67
May 31 20:14:05 laptop3 dhclient: DHCPACK from 172.16.97.254
May 31 20:14:05 laptop3 dhclient: bound to 172.16.97.100 -- renewal in 41631
seconds.Each entry consists of a date, time, hostname (laptop3 in this example), program name or other prefix (dhclient), and a text message. Note that the facility and priority are not recorded in the logfile. Since the /var/log/message file can be very large, it's worthwhile using a tool such as grep to search for specific records. For example, you can view all of the kernel messages with the command: $ grep kernel /var/log/messages
May 30 04:23:08 bluesky kernel: SELinux: initialized (dev hdd, type iso9660),
uses genfs_contexts
May 31 20:48:40 bluesky kernel: atkbd.c: Unknown key pressed (translated
set 2, code 0x85 on isa0060/serio0).
May 31 20:48:40 bluesky kernel: atkbd.c: Use 'setkeycodes e005 <keycode>'
to make it known.
May 31 21:14:54 bluesky kernel: cdrom: This disc doesn't have any tracks I
recognize!8.7.1.2. Creating your own logfile entriesYou can generate syslog messages using the logger command-line tool. Simply provide your text as arguments: $ logger Added host lightning to /etc/hosts The message recorded in /var/log/messages contains the username as the prefix: Jun 1 02:32:59 darkday chris: Added host lightning to /etc/hosts It's convenient to log information about changes you have made on the system in this way, entering them as you work. Your notes will be interleaved with system-generated log messages, making it easy to see the relationship between the changes that you have made and any messages that start or stop appearing in the log as a result. By default, logger uses the facility user and the priority notice. You can override this using the -p option, and you can override the insertion of the username by supplying an alternate tag with the -t option: $ logger -p local1.crit -t cooling Stopped water pump Which would result in this message being logged: Jun 1 09:54:49 darkday cooling: Stopped water pump An alias can be used to simplify logging from the command line: $ alias note='logger -p local4.notice' $ note Ran yum update If you are logging a message that contains metacharacters, surround the message with quotation marks. By adding a custom rule to /etc/syslog.conf, the messages sent to the local1 facility can be placed in their own file (in addition to being logged in /var/log/messages): local1.* /var/log/cooling The security context of any new logfiles must be set to the same context as /var/log/messages: # touch /var/log/cooling # ls -Z /var/log/messages /var/log/cooling -rw-r--r-- root root user_u:object_r:var_log_t /var/log/cooling -rw------- root root system_u:object_r:var_log_t /var/log/messages # chcon system_u:object_r:var_log_t /var/log/cooling # chmod 0600 /var/log/cooling # Optional! # ls -Z /var/log/messages /var/log/cooling -rw------- root root system_u:object_r:var_log_t /var/log/cooling -rw------- root root system_u:object_r:var_log_t /var/log/messages 8.7.1.3. Keeping an eye on logsThe -f option to tail provides a convenient way to watch messages that are being appended to a file and is perfect for use with logfiles: # tail -f /var/log/messages Jun 1 08:47:14 darkday kernel: hub 1-0:1.0: over-current change on port 1 Jun 1 08:47:14 darkday kernel: hub 1-0:1.0: port 2 disabled by hub (EMI?), re-enabling... Jun 1 08:47:14 darkday kernel: hub 1-0:1.0: over-current change on port 2 Jun 1 08:47:14 darkday kernel: usb 1-2: USB disconnect, address 4 Jun 1 08:47:14 darkday kernel: usb 1-2: new low speed USB device using uhci_hcd and address 5 Jun 1 08:47:14 darkday kernel: usb 1-2: configuration #1 chosen from 1 choice Jun 1 08:47:14 darkday kernel: input: Logitech USB-PS/2 Optical Mouse as /class/input/input4 Jun 1 08:47:14 darkday kernel: input: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:1f.2-2 Jun 1 09:54:49 darkday cooling: Water temperature exceeds 70C Jun 1 09:54:49 darkday cooling: Water temperature exceeds 85C ...(Additional lines are displayed as they are added to the logfile)...
This tail command will display the last 10 lines in the file, and then additional lines within a second of the time that they are appended to the file. It can be left running in a terminal window in the corner of the screen while you perform system administration tasks. 8.7.1.4. Configuring remote loggingThe syslog service was designed to facilitate remote logging. This is very useful in two circumstances:
To configure a syslog network server, edit that host's /etc/sysconfig/syslog file, which initially looks like this: # Options to syslogd # -m 0 disables 'MARK' messages. # -r enables logging from remote machines # -x disables DNS lookups on messages recieved with -r # See syslogd(8) for more details SYSLOGD_OPTIONS="-m 0" # Options to klogd # -2 prints all kernel oops messages twice: once for klogd to decode, and # once for processing with 'ksymoops' # -x disables all klogd processing of oops messages entirely # See klogd(8) for more details KLOGD_OPTIONS="-x" # SYSLOG_UMASK=077 # set this to a umask value to use for all logfiles, as in umask(1). # By default, all permissions are removed for "group" and "other". Change the SYSLOGD_OPTIONS line to include -r (remote logging): SYSLOGD_OPTIONS="-m 0 -r" Then restart syslogd: # service syslog restart Shutting down kernel logger: [ OK ] Shutting down system logger: [ OK ] Starting system logger: [ OK ] Starting kernel logger: [ OK ]
Next, edit the file /etc/syslog.conf on the machines that will be forwarding log messages to the syslog server, and add this line: *.* @syslogserver
This will forward all messages to the remote host syslogserver (which may be an IP address or hostname). Restart syslogd to activate the changes.
The result will be a combined log containing entries from both the syslog server and the host that is forwarding its log messages: Jun 1 02:52:33 darkday named[13255]: starting BIND 9.3.2 -u named Jun 1 02:52:33 darkday named[13255]: found 1 CPU, using 1 worker thread Jun 1 02:52:33 darkday named[13255]: loading configuration from '/etc/named.conf' Jun 1 02:52:33 darkday named[13255]: listening on IPv4 interface lo, 127.0.0.1#53 Jun 1 02:52:33 darkday named[13255]: listening on IPv4 interface eth0, 172.16.97.100#53 Jun 1 02:52:33 darkday named[13255]: command channel listening on 127.0.0.1#953 Jun 1 02:52:33 darkday named[13255]: zone 0.in-addr.arpa/IN: loaded serial 42 Jun 1 02:52:33 darkday named[13255]: zone 0.0.127.in-addr.arpa/IN: loaded serial 1997022700 Jun 1 02:52:33 darkday named[13255]: zone 255.in-addr.arpa/IN: loaded serial 42 Jun 1 02:52:33 darkday named[13255]: zone 0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 1997022700 Jun 1 02:52:33 darkday named[13255]: zone localdomain/IN: loaded serial 42 Jun 1 02:52:33 darkday named[13255]: zone localhost/IN: loaded serial 42 Jun 1 02:52:33 darkday named[13255]: running Jun 1 02:57:22 bluesky chris: VNC service configured, restarting xinetd Jun 1 02:57:29 bluesky xinetd[15394]: Exiting... Jun 1 02:57:29 bluesky xinetd[15452]: xinetd Version 2.3.13 started with libwrap loadavg options compiled in. Jun 1 02:57:29 bluesky xinetd[15452]: Started working: 1 available service Notice that this log contains entries from darkday (the syslog server) as well as from bluesky (which is forwarding log messages to darkday). Notice also the system administrator's note on bluesky, stating the reason that xinetd was being restarted.
8.7.1.5. Automated log watchingThere's not much point in collecting all this information if the logs are never read, but reading logfiles is boring, tedious work. Fortunately, the logwatch package automates this process, sending a daily summary email to alert you to important log entries. The daily summary is emailed to root on the local machine. Email to the root user should be redirected to a specific user or users by the /etc/aliases file. Edit this file and uncomment the entry for root found at the the end, inserting the name of a user who is responsible for administering the system (or a list of people separated by commas). In this example, all mail for root is redirected to chris@fedorabook.com: # Person who should get root's mail
root:
chris@fedorabook.comHere is a typical daily logwatch summary: From: root <root@bluesky.fedorabook.com>
To: root@bluesky.fedorabook.com
Subject: LogWatch for bluesky.fedorabook.com
Date: Wed, 31 May 2006 04:02:17 -0400
################### LogWatch 7.1 (11/12/05) ####################
Processing Initiated: Thu Jun 1 02:52:14 2006
Date Range Processed: yesterday
( 2006-May-31 )
Period is day.
Detail Level of Output: 10
Type of Output: unformatted
Logfiles for Host: bluesky.fedorabook.com
##################################################################
--------------------- httpd Begin ------------------------
A total of 3 unidentified 'other' records logged
GET /level/16/exec/-///pwd HTTP/1.0 with response code(s)
2 404 responses
POST /garethjones/photos/--WEBBOT-SELF-- HTTP/1.0 with response code(s)
1 404 responses
GET http://bluesky.fedorabook.com/foo HTTP/1.1 with response code(s)
1 404 responses
---------------------- httpd End -------------------------
--------------------- SSHD Begin ------------------------
Users logging in through sshd:
chris:
172.16.97.2: 3 times
--------------------- SSHD End -------------------------
--------------------- Disk Space Begin ------------------------
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/main-root 9.5G 2.9G 6.1G 33% /
/dev/hda1 99M 9.7M 84M 11% /boot
/dev/mapper/main-home 4.9G 24M 4.7G 1% /home
---------------------- Disk Space End -------------------------
###################### LogWatch End ######################### This report will vary according to the services you have installed, but it provides a simple, easy-to-scan summary of log entries that may warrant attention. It also provides a summary of free disk space; if you methodically review these email messages, you won't be caught unaware when your storage needs start to inch upward. 8.7.1.6. Log rotationLogfiles can grow to be massive. The Fedora logrotate package automatically moves historical log data into history files and keeps a limited number of history files on hand. logrotate is configured through the master configuration file /etc/logrotate.conf: # see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) logfiles after rotating old ones
create
# uncomment this if you want your logfiles compressed
#compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
rotate 1
}
# system-specific logs may be also be configured here.The most frequently altered lines are highlighted in bold: logrotate is initially configured to rotate logs every week and to save the last four historical logfiles in addition to the current log. If you have a lot of storage and wish to keep more history, edit the rotate line to increase the number of history files maintained, or change the weekly line to monthly to reduce the frequency of history snapshots (which can make it easier to analyze patterns over a longer period of time without merging data from several files). The default configuration results in five separate message files being present on the system: $ ls -l /var/log/messages* -rw------- 1 root root 86592 Jun 1 02:49 /var/log/messages -rw------- 1 root root 85053 May 30 02:03 /var/log/messages.1 -rw------- 1 root root 105491 May 26 23:51 /var/log/messages.2 -rw------- 1 root root 74062 May 7 04:12 /var/log/messages.3 -rw------- 1 root root 286194 May 2 13:00 /var/log/messages.4
8.7.2. How Does It Work?The main system logging utility is named syslog. It is network-based and uses a server daemon, syslogd, which receives messages from all sorts of system programs through the Unix domain socket /var/log. These messages are matched against the lines in /etc/syslog.conf and written to the selected destinations. Kernel messages are stored in a buffer that is read by a helper daemon named klogd, either by reading the file /proc/kmesg or by using a kernel system call. klogd then forwards these messages to syslogd for inclusion in the system logs. A syslog network server listens to UDP port 514 and processes any messages received there through the normal routing decisions. One significant problem with the syslog implementation is that there is absolutely no authentication performed. Any application can log any message with any facility and priority. Therefore it is relatively easy to spoof log messages or to create a denial-of-service attack by sending huge numbers of logfile entries, eventually filling all available disk space and making it impossible to log further events. (For this reason, it is a good idea to use a separate filesystem for /var/log). The logwatch and logrotate programs are activated by cron through their entries in /etc/cron.daily. 8.7.3. What About...8.7.3.1. ...sending log messages to a program?The standard Fedora syslog program does not support output to a program such as a mailer. However, you can easily write a script that reads a logfile using the tail command and outputs new log entries to a program. This example emails log messages to a pager or cell phone text service: #!/bin/bash
DESTINATION=8885551234@pagercompany.example.com
tail -0f /var/log/messages|
while read LINE
do
echo $LINE|
mail $DESTINATION
doneTo use this script, place it in the file /usr/local/bin/log-mail and add read and execute permissions: # chmod u+rx /usr/local/bin/log-mail
# log-mail
You can also have log output read to you over the system's speakers: #!/bin/bash
logger -t log-speak "Starting log reading."
sleep 0.3
tail -1f /var/log/messages|
while read LINE
do
# The sed expressions remove the date/time and PIDs
# from messages to shorten the text.
echo $LINE|
sed -e "s/^.\{17\}[^ ]*//" \
-e "s/\[.*\]//g"|
festival --tts
done8.7.3.2. ...outputting to a named pipe?A named pipe is a special type of file that can be used to pass messages between two programs. While syslog supports writing to named pipes, the default SELinux security policy prohibits it. To output to a named pipe, you must first disable SELinux protection for syslogd by setting the syslogd_disable_trans boolean and then create the named pipe with mkfifo: # setsebool -P syslogd_disable_trans=1 # mkfifo /var/log/messagepipe Next, create an entry in /etc/syslog.conf, placing a pipe symbol in front of the destination pathname: *.* |/var/log/messagepipe Restart syslogd. You can then follow the message output with a simple file read: # service syslog restart Shutting down kernel logger: [ OK ] Shutting down system logger: [ OK ] Starting system logger: [ OK ] Starting kernel logger: [ OK ] # cat /var/log/messagepipe ...(Messages appear as they are logged)... 8.7.3.3. ...logging messages from printers, routers, and other network devices?Most network hardware offers the option of logging messages to a syslog server. Simply enter the IP address of your syslog network server into the configuration settings of the device. 8.7.3.4. ...using patterns within the message text to determine message routing?The syslog-ng package from Fedora Extras can be used in place of the standard syslogd and klogd programs. It uses a different configuration file syntax, and it supports message-text matching and message routing to programs.
8.7.4. Where Can I Learn More?
 |
|
|
